2025-12-19 16:02:52.214 INFO 3588644 --- [prometheus-minute-schedule-1,3cebcc69c08c2ab0,a3c7d6ce5ebc30a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-12-19 16:02:52.816 INFO 3588644 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-12-19 16:02:52.930 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-2,cdd44e71233c5e4c,2fef69c8bd7ca4f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=cdd44e71233c5e4c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:02:52.930 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-2,cdd44e71233c5e4c,2fef69c8bd7ca4f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=cdd44e71233c5e4c, duration=1 ms] 2025-12-19 16:02:53.071 INFO 3588644 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 172.18.96.94:8080 refreshes policies... 2025-12-19 16:02:53.072 INFO 3588644 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-12-19 16:02:53.175 WARN 3588644 --- [ocp-async-1,c0fbc365acf33989,657a0bcb678a4a15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:02:53.388 INFO 3588644 --- [ocp-async-5,7b7fa35e9fe8c040,cfc01f1936b79163] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:02:53.987 INFO 3588644 --- [alarm-task-1,7e7b76990905e0c5,8ad399334d48fd6d] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-12-19T16:02:53.986097+08:00, lastProcessTime=2025-12-19T16:02:43.987926+08:00 2025-12-19 16:02:54.056 INFO 3588644 --- [alarm-task-1,7e7b76990905e0c5,8ad399334d48fd6d] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-12-19T16:02:53.986097+08:00 2025-12-19 16:02:55.175 INFO 3588644 --- [ocp-schedules-16,0e0182a1e75a4f38,a8037f58bee7670e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-12-19 16:02:55.175 WARN 3588644 --- [ocp-async-5,b99cb571e82db73c,0433f642b6553614] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:02:55.210 INFO 3588644 --- [prometheus-minute-schedule-1,cfdbb4d8a46b6e39,8b22ff0f7a34a447] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1766131380, interval=60 2025-12-19 16:02:55.214 INFO 3588644 --- [prometheus-minute-schedule-1,cfdbb4d8a46b6e39,8b22ff0f7a34a447] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-12-19 16:02:55.389 INFO 3588644 --- [ocp-async-8,e51fd6f3feb299c4,4e6c46ab3f947c4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:02:55.944 INFO 3588644 --- [ocp-schedules-15,14343e1d635c1d1d,fd17e2f3b56314b0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-12-19 16:02:55.949 INFO 3588644 --- [ocp-schedules-15,14343e1d635c1d1d,fd17e2f3b56314b0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[879d333a-5c02-4a99-a8f8-c9f82dd8e625] 2025-12-19 16:02:55.950 INFO 3588644 --- [ocp-schedules-15,14343e1d635c1d1d,fd17e2f3b56314b0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[879d333a-5c02-4a99-a8f8-c9f82dd8e625] 2025-12-19 16:02:56.072 INFO 3588644 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 172.18.96.94:8080 refreshes policies... 2025-12-19 16:02:56.073 INFO 3588644 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-12-19 16:02:56.272 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-4,cdf553c6224c5c13,e9cfc55448d61701] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=cdf553c6224c5c13, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:02:56.272 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-4,cdf553c6224c5c13,e9cfc55448d61701] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=cdf553c6224c5c13, duration=0 ms] 2025-12-19 16:02:57.175 WARN 3588644 --- [ocp-async-2,696b93186aa61010,686bb42ad9bcefd6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:02:57.271 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-5,b838e8190d2c188c,fc8a801be13b27e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=b838e8190d2c188c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:02:57.272 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-5,b838e8190d2c188c,fc8a801be13b27e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=b838e8190d2c188c, duration=1 ms] 2025-12-19 16:02:57.389 INFO 3588644 --- [ocp-async-6,e01891eeb24a235b,5f677fea4527a492] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:02:57.949 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-8,cb3f9352683c707c,a85c39d84bd1f256] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=cb3f9352683c707c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:02:57.949 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-8,cb3f9352683c707c,a85c39d84bd1f256] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=cb3f9352683c707c, duration=1 ms] 2025-12-19 16:02:58.076 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-1,059b31b22ee73397,dd9310ce2e6b4b11] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=059b31b22ee73397, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:02:58.077 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-1,059b31b22ee73397,dd9310ce2e6b4b11] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=059b31b22ee73397, duration=1 ms] 2025-12-19 16:02:58.089 INFO 3588644 --- [ocp-schedules-14,13c6122bbc9bec39,6ea2dc18bc13d136] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-12-19 16:02:58.162 WARN 3588644 --- [scanner-137160191-1,13c6122bbc9bec39,c9fed0192a711c3c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 13 2025-12-19 16:02:58.210 INFO 3588644 --- [prometheus-minute-schedule-1,a2a28bd571de8947,6ce1dd08623af1f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1766131380, interval=60 2025-12-19 16:02:58.225 INFO 3588644 --- [prometheus-minute-schedule-1,a2a28bd571de8947,6ce1dd08623af1f1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-12-19 16:02:58.340 INFO 3588644 --- [alarm-detect-10,13c6122bbc9bec39,5847dd2b0dac018e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_install_disk_percent_over_threshold detect time range 1766131350 ~ 1766131368 matched, alarm-level: 1, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_install_disk_usage_percent=[SeriesMetricValues(metric=Metric{name='ob_install_disk_usage_percent', labels={app=HOST, host=172.18.65.35, host_name=test-all-ob-observer-2, install_path=/data/oceanbase, mount_point=/data, ob_cluster_id=1761559196, ob_cluster_name=test_all_ob_cluster, obzone=zone2, svr_ip=172.18.65.35}}, data=[ValueData(timestamp=1766131365, value=97.943)])]}, metricEvaluateResults={ob_install_disk_usage_percent > 97=[ExpressionEvaluateResult(matchValue=97.0, metric=ob_install_disk_usage_percent, match=true, value=97.943, values={1766131365=97.943}, valueCount=1, labels={app=HOST, host=172.18.65.35, host_name=test-all-ob-observer-2, install_path=/data/oceanbase, mount_point=/data, ob_cluster_id=1761559196, ob_cluster_name=test_all_ob_cluster, obzone=zone2, svr_ip=172.18.65.35})]}, matched=true, isContainValidValue=null, message=null, level=1) 2025-12-19 16:02:58.340 INFO 3588644 --- [alarm-detect-10,13c6122bbc9bec39,5847dd2b0dac018e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_install_disk_percent_over_threshold detect time range 1766131350 ~ 1766131368 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_install_disk_usage_percent=[SeriesMetricValues(metric=Metric{name='ob_install_disk_usage_percent', labels={app=HOST, host=172.18.65.35, host_name=test-all-ob-observer-2, install_path=/data/oceanbase, mount_point=/data, ob_cluster_id=1761559196, ob_cluster_name=test_all_ob_cluster, obzone=zone2, svr_ip=172.18.65.35}}, data=[ValueData(timestamp=1766131365, value=97.943)])]}, metricEvaluateResults={ob_install_disk_usage_percent > 95=[ExpressionEvaluateResult(matchValue=95.0, metric=ob_install_disk_usage_percent, match=true, value=97.943, values={1766131365=97.943}, valueCount=1, labels={app=HOST, host=172.18.65.35, host_name=test-all-ob-observer-2, install_path=/data/oceanbase, mount_point=/data, ob_cluster_id=1761559196, ob_cluster_name=test_all_ob_cluster, obzone=zone2, svr_ip=172.18.65.35})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-12-19 16:02:58.890 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-2,9cfcc88b7c9b67fb,4799d80410357bca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=9cfcc88b7c9b67fb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:02:58.890 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-2,9cfcc88b7c9b67fb,4799d80410357bca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=9cfcc88b7c9b67fb, duration=0 ms] 2025-12-19 16:02:59.073 INFO 3588644 --- [pool-13-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 172.18.96.94:8080 refreshes policies... 2025-12-19 16:02:59.074 INFO 3588644 --- [pool-13-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-12-19 16:02:59.175 WARN 3588644 --- [ocp-async-6,a289395480358879,00b39cef22e10f25] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:02:59.389 INFO 3588644 --- [ocp-async-7,cdafffefe0557f38,2a65561fd3c4fd10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:02:59.831 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-4,18614d4111c70567,afbf798bd42186be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=18614d4111c70567, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:02:59.831 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-4,18614d4111c70567,afbf798bd42186be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=18614d4111c70567, duration=1 ms] 2025-12-19 16:03:00.527 INFO 3588644 --- [ocp-schedules-8,a093453cdcc252b3,1c9f087b7f22338e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-12-19 16:03:00.529 INFO 3588644 --- [ocp-schedules-8,a093453cdcc252b3,1c9f087b7f22338e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-12-19 16:03:00.543 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-5,a440d06d4c8f0300,948ac8c0bccf5b54] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.25, traceId=a440d06d4c8f0300, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:03:00.543 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-5,a440d06d4c8f0300,948ac8c0bccf5b54] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.25, traceId=a440d06d4c8f0300, duration=0 ms] 2025-12-19 16:03:00.637 INFO 3588644 --- [ocp-async-8,56dcb9578af96422,7fa9bacbbb5f86bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-12-19 16:03:00.814 INFO 3588644 --- [subtask-executor31,75f534a7064b4a97,fb46c94405919c04] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1761559196,test_all_ob_cluster,1002,1766102400000000,2000,30000000 2025-12-19 16:03:00.818 INFO 3588644 --- [subtask-executor31,75f534a7064b4a97,fb46c94405919c04] c.o.o.perf.sql.internal.SqlParseService : parse 48 sqls for tenant:ObTenantEntity{id=5, name=test, obTenantId=1002, status=NORMAL} 2025-12-19 16:03:01.175 WARN 3588644 --- [ocp-async-3,12236bce0ea82ad9,894951c1425b163e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:03:01.210 INFO 3588644 --- [prometheus-minute-schedule-1,3f43f72d71da3ab9,4f76f8066d1816c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1766131440, interval=60 2025-12-19 16:03:01.214 INFO 3588644 --- [prometheus-minute-schedule-1,3f43f72d71da3ab9,4f76f8066d1816c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-12-19 16:03:01.389 INFO 3588644 --- [ocp-async-8,456a29e1aa55fdfe,f95e57d498821798] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:03:01.509 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-8,65059d0189e5085d,24e0bc45f3971215] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=65059d0189e5085d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:03:01.509 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-8,65059d0189e5085d,24e0bc45f3971215] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=65059d0189e5085d, duration=1 ms] ^C [root@test_ocp_ob_cluster log]# ll total 52824 -rw-r--r-- 1 root root 14045553 Dec 11 16:51 bootstrap.log -rw-r--r-- 1 root root 40027955 Dec 19 16:08 ocp-server.log drwxr-xr-x 2 root root 4096 Dec 11 16:48 sandbox drwxr-xr-x 2 root root 4096 Dec 19 00:11 security [root@test_ocp_ob_cluster log]# tail -f ocp-server.log 2025-12-19 16:08:38.519 INFO 3588644 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=13768 2025-12-19 16:08:38.524 INFO 3588644 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm odp_sql_query_slow null 2025-12-19 16:08:38.539 INFO 3588644 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentAlarmManagerImpl : create alarm odp_sql_execute_failed null 2025-12-19 16:08:38.539 INFO 3588644 --- [ocp-schedules-6,4eef8360d23b03c0,31fcfe1ff9dcabf6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-12-19 16:08:38.634 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-3,2ced34901365be57,b7c534fcf4c0b7b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=2ced34901365be57, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:08:38.635 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-3,2ced34901365be57,b7c534fcf4c0b7b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=2ced34901365be57, duration=1 ms] 2025-12-19 16:08:39.202 WARN 3588644 --- [ocp-async-10,9a2f5c0eae70f119,be352b86697b6776] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:08:39.419 INFO 3588644 --- [ocp-async-8,cfaa6aaa598149a5,ed148c25bf2eedd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:08:39.451 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-6,146e9a99d8886c9b,985d773b13aa8f68] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=146e9a99d8886c9b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:08:39.451 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-6,146e9a99d8886c9b,985d773b13aa8f68] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=146e9a99d8886c9b, duration=0 ms] 2025-12-19 16:08:40.002 INFO 3588644 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-12-19 16:08:40.002 INFO 3588644 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-12-19 16:08:40.014 INFO 3588644 --- [Check obproxy1,3cf13bc4b08eb7b2,9d670cdff8d700cc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.18.64.252 2883 2025-12-19 16:08:40.014 INFO 3588644 --- [Collect obproxy1,20b2ea63bf85a978,c02ab3273db88e06] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.18.64.252 2883 2025-12-19 16:08:40.014 INFO 3588644 --- [Check obproxy1,3cf13bc4b08eb7b2,9d670cdff8d700cc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-12-19 16:08:40.014 INFO 3588644 --- [Collect obproxy1,20b2ea63bf85a978,c02ab3273db88e06] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-12-19 16:08:40.041 INFO 3588644 --- [Collect obproxy1,20b2ea63bf85a978,c02ab3273db88e06] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-12-19 16:08:40.043 INFO 3588644 --- [Check obproxy1,3cf13bc4b08eb7b2,9d670cdff8d700cc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_id%] 2025-12-19 16:08:40.054 INFO 3588644 --- [Check obproxy1,3cf13bc4b08eb7b2,9d670cdff8d700cc] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 2 success, update last active time to now 2025-12-19 16:08:40.054 INFO 3588644 --- [Collect obproxy1,20b2ea63bf85a978,c02ab3273db88e06] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-12-19 16:08:40.054 INFO 3588644 --- [Check obproxy1,3cf13bc4b08eb7b2,9d670cdff8d700cc] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 2 proxy_id: 2, need to sync to meta: false 2025-12-19 16:08:40.064 INFO 3588644 --- [Collect obproxy1,20b2ea63bf85a978,c02ab3273db88e06] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-12-19 16:08:40.209 INFO 3588644 --- [ocp-schedules-19,37102cabdc1ba9b1,de3da628023ea792] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-12-19T16:08:40.209340314+08:00, clusters=[1, 2] 2025-12-19 16:08:40.210 INFO 3588644 --- [prometheus-minute-schedule-1,8949b65a6c54c2d0,02684c46f768acb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1766131740, interval=60 2025-12-19 16:08:40.214 INFO 3588644 --- [prometheus-minute-schedule-1,8949b65a6c54c2d0,02684c46f768acb1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-12-19 16:08:40.223 INFO 3588644 --- [ocp-schedules-19,37102cabdc1ba9b1,de3da628023ea792] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for test_ocp_ob_cluster, save 3 to SqlAggregateData 2025-12-19 16:08:40.336 INFO 3588644 --- [ocp-schedules-19,37102cabdc1ba9b1,de3da628023ea792] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for test_all_ob_cluster, save 35 to SqlAggregateData 2025-12-19 16:08:40.402 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-4,da14fa3bdd472b01,fc924c0b11a1a87a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=da14fa3bdd472b01, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:08:40.402 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-4,da14fa3bdd472b01,fc924c0b11a1a87a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=da14fa3bdd472b01, duration=0 ms] 2025-12-19 16:08:40.642 INFO 3588644 --- [ocp-async-7,e7ff1bd936f91f02,2d30607d3850dc49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-12-19 16:08:40.829 INFO 3588644 --- [ocp-schedules-4,c8cdff4a95fa7494,8a6cad267012a1d5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-12-19 16:08:40.831 INFO 3588644 --- [ocp-schedules-4,c8cdff4a95fa7494,8a6cad267012a1d5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-12-19 16:08:41.135 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-8,f63320da175a1367,8eae0462f830ffca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.25, traceId=f63320da175a1367, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:08:41.135 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-8,f63320da175a1367,8eae0462f830ffca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.25, traceId=f63320da175a1367, duration=1 ms] 2025-12-19 16:08:41.189 INFO 3588644 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 172.18.96.94:8080 refreshes policies... 2025-12-19 16:08:41.190 INFO 3588644 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-12-19 16:08:41.202 WARN 3588644 --- [ocp-async-3,70c0819e05aeb6a8,36010c977798fe2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:08:41.419 INFO 3588644 --- [ocp-async-7,eef1c59a63371509,ea3e6f64cdd34aae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:08:41.788 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/backup/config/storages/test, client 183.6.153.90, traceId 5f1d3d26073f9acf] 2025-12-19 16:08:41.799 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/backup/config/storages/test?null, client=183.6.153.90, traceId=5f1d3d26073f9acf, method=NoDataResponse com.oceanbase.ocp.server.common.controller.ObBackupController.testStorageConfig(TestStorageParam), args=TestStorageParam(hostIdList=null, clusterId=2, obBackupStorageBaseInfo=ObBackupStorageBaseInfo(backupStorageType=BACKUP_STORAGE_S3, storageUrl=kungeek-oceanbase-******/test_all_ob_cluster_inventory/test/backupsets, ossAccessKey=null, cosAccessKey=null, obsAccessKey=null, s3AccessKey=S3AccessKey(region=ap-beijing, endpoint=cos.ap-beijing.myqcloud.com, accessKeyId=xxx), customBackupRangeList=[], advancedConfigList=[BackupAdvancedConfigInfo(name=delete_mode, values=[delete]), BackupAdvancedConfigInfo(name=checksum_type, values=[md5]), BackupAdvancedConfigInfo(name=addressing_model, values=[virtual_hosted_style])]), backupPurposeType=BACKUP, storageId=null),] 2025-12-19 16:08:41.800 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.b.i.f.c.BackupCommonServiceImpl : test storage config, type=s3, url=kungeek-oceanbase-******/test_all_ob_cluster_inventory/test/backupsets 2025-12-19 16:08:41.810 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-12-19 16:08:41.811 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-12-19 16:08:41.818 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.23:62888/api/v1/backup/storage/test, request body:TestBackupStorageRequest(backupStorageType=s3, bucketName=kungeek-oceanbase-******, objectName=test_all_ob_cluster_inventory/test/backupsets, accessDomainName=https://cos.ap-beijing.myqcloud.com, region=ap-beijing, accessKeyId=******), params:null 2025-12-19 16:08:41.856 WARN 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.b.i.d.f.c.RemoteBackupFileParser : class com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser$$Lambda$6801/0x0000146d26637890 com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:check storage bucket failed, bucket: kungeek-oceanbase-******, reason: operation error S3: HeadBucket, https response error StatusCode: 403, RequestID: Njk0NTA4MDlfNjViNWJhMDlfZjA4OF8yOGYzMWEwNA==, HostID: , api error Forbidden: Forbidden at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:479) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:289) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:117) at com.oceanbase.ocp.executor.executor.AgentExecutor.testBackupStorage(AgentExecutor.java:773) at com.oceanbase.ocp.backup.internal.agent.BackupHostOperationService.testBackupStorage(BackupHostOperationService.java:347) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.lambda$testStorage$0(RemoteBackupFileParser.java:47) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.templateOfRemoteAgentOperation(RemoteBackupFileParser.java:36) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.testStorage(RemoteBackupFileParser.java:46) at com.oceanbase.ocp.backup.internal.utils.ObBackupChecker.checkStorageAccessible(ObBackupChecker.java:469) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl.testStorageConfig(BackupCommonServiceImpl.java:648) at jdk.internal.reflect.GeneratedMethodAccessor3990.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:724) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl$$SpringCGLIB$$0.testStorageConfig() at com.oceanbase.ocp.server.common.controller.ObBackupController.testStorageConfig(ObBackupController.java:227) at jdk.internal.reflect.GeneratedMethodAccessor3469.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:132) at jdk.internal.reflect.GeneratedMethodAccessor447.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:642) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:632) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.proceed(AuthorizationManagerBeforeMethodInterceptor.java:268) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.attemptAuthorization(AuthorizationManagerBeforeMethodInterceptor.java:263) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:196) at org.springframework.security.config.annotation.method.configuration.DeferringMethodInterceptor.invoke(DeferringMethodInterceptor.java:44) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:728) at com.oceanbase.ocp.server.common.controller.ObBackupController$$SpringCGLIB$$0.testStorageConfig() at jdk.internal.reflect.GeneratedMethodAccessor3469.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231) at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340) at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128) at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:101) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:50) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.saml2.provider.service.web.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:198) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:131) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$4(HandlerMappingIntrospector.java:267) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:238) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1769) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang.Thread.run(Unknown Source) 2025-12-19 16:08:41.858 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-12-19 16:08:41.859 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-12-19 16:08:41.867 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.35:62888/api/v1/backup/storage/test, request body:TestBackupStorageRequest(backupStorageType=s3, bucketName=kungeek-oceanbase-******, objectName=test_all_ob_cluster_inventory/test/backupsets, accessDomainName=https://cos.ap-beijing.myqcloud.com, region=ap-beijing, accessKeyId=******), params:null 2025-12-19 16:08:41.905 WARN 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.b.i.d.f.c.RemoteBackupFileParser : class com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser$$Lambda$6801/0x0000146d26637890 com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:check storage bucket failed, bucket: kungeek-oceanbase-******, reason: operation error S3: HeadBucket, https response error StatusCode: 403, RequestID: Njk0NTA4MDlfY2U1NjYwMGJfZGU1OF9kMzcwYmMx, HostID: , api error Forbidden: Forbidden at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:479) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:289) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:117) at com.oceanbase.ocp.executor.executor.AgentExecutor.testBackupStorage(AgentExecutor.java:773) at com.oceanbase.ocp.backup.internal.agent.BackupHostOperationService.testBackupStorage(BackupHostOperationService.java:347) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.lambda$testStorage$0(RemoteBackupFileParser.java:47) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.templateOfRemoteAgentOperation(RemoteBackupFileParser.java:36) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.testStorage(RemoteBackupFileParser.java:46) at com.oceanbase.ocp.backup.internal.utils.ObBackupChecker.checkStorageAccessible(ObBackupChecker.java:469) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl.testStorageConfig(BackupCommonServiceImpl.java:648) at jdk.internal.reflect.GeneratedMethodAccessor3990.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:724) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl$$SpringCGLIB$$0.testStorageConfig() at com.oceanbase.ocp.server.common.controller.ObBackupController.testStorageConfig(ObBackupController.java:227) at jdk.internal.reflect.GeneratedMethodAccessor3469.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:132) at jdk.internal.reflect.GeneratedMethodAccessor447.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:642) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:632) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.proceed(AuthorizationManagerBeforeMethodInterceptor.java:268) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.attemptAuthorization(AuthorizationManagerBeforeMethodInterceptor.java:263) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:196) at org.springframework.security.config.annotation.method.configuration.DeferringMethodInterceptor.invoke(DeferringMethodInterceptor.java:44) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:728) at com.oceanbase.ocp.server.common.controller.ObBackupController$$SpringCGLIB$$0.testStorageConfig() at jdk.internal.reflect.GeneratedMethodAccessor3469.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231) at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340) at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128) at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:101) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:50) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.saml2.provider.service.web.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:198) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:131) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$4(HandlerMappingIntrospector.java:267) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:238) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1769) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang.Thread.run(Unknown Source) 2025-12-19 16:08:41.907 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-12-19 16:08:41.908 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-12-19 16:08:41.916 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.64.87:62888/api/v1/backup/storage/test, request body:TestBackupStorageRequest(backupStorageType=s3, bucketName=kungeek-oceanbase-******, objectName=test_all_ob_cluster_inventory/test/backupsets, accessDomainName=https://cos.ap-beijing.myqcloud.com, region=ap-beijing, accessKeyId=******), params:null 2025-12-19 16:08:41.955 WARN 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.b.i.d.f.c.RemoteBackupFileParser : class com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser$$Lambda$6801/0x0000146d26637890 com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:check storage bucket failed, bucket: kungeek-oceanbase-******, reason: operation error S3: HeadBucket, https response error StatusCode: 403, RequestID: Njk0NTA4MDlfNWEwZWZmMWVfMTM5NjZfMWVjYTUzNmE=, HostID: , api error Forbidden: Forbidden at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:479) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:289) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:117) at com.oceanbase.ocp.executor.executor.AgentExecutor.testBackupStorage(AgentExecutor.java:773) at com.oceanbase.ocp.backup.internal.agent.BackupHostOperationService.testBackupStorage(BackupHostOperationService.java:347) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.lambda$testStorage$0(RemoteBackupFileParser.java:47) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.templateOfRemoteAgentOperation(RemoteBackupFileParser.java:36) at com.oceanbase.ocp.backup.internal.data.file.common.RemoteBackupFileParser.testStorage(RemoteBackupFileParser.java:46) at com.oceanbase.ocp.backup.internal.utils.ObBackupChecker.checkStorageAccessible(ObBackupChecker.java:469) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl.testStorageConfig(BackupCommonServiceImpl.java:648) at jdk.internal.reflect.GeneratedMethodAccessor3990.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:724) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl$$SpringCGLIB$$0.testStorageConfig() at com.oceanbase.ocp.server.common.controller.ObBackupController.testStorageConfig(ObBackupController.java:227) at jdk.internal.reflect.GeneratedMethodAccessor3469.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:132) at jdk.internal.reflect.GeneratedMethodAccessor447.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:642) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:632) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.proceed(AuthorizationManagerBeforeMethodInterceptor.java:268) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.attemptAuthorization(AuthorizationManagerBeforeMethodInterceptor.java:263) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:196) at org.springframework.security.config.annotation.method.configuration.DeferringMethodInterceptor.invoke(DeferringMethodInterceptor.java:44) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:728) at com.oceanbase.ocp.server.common.controller.ObBackupController$$SpringCGLIB$$0.testStorageConfig() at jdk.internal.reflect.GeneratedMethodAccessor3469.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231) at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340) at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128) at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:101) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:50) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.saml2.provider.service.web.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:198) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:131) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$4(HandlerMappingIntrospector.java:267) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:238) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1769) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang.Thread.run(Unknown Source) 2025-12-19 16:08:41.955 ERROR 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.b.internal.utils.ObBackupChecker : test storage config failed, directory not exists: s3://kungeek-oceanbase-******/test_all_ob_cluster_inventory/test/backupsets 2025-12-19 16:08:41.961 ERROR 3588644 --- [http-nio-0.0.0.0-8080-exec-7,5f1d3d26073f9acf,4f27fc8c8dec8fbb] c.o.o.s.c.trace.RequestTracingAspect : Unexpected API Error: [POST /api/v2/backup/config/storages/test client=183.6.153.90, traceId=5f1d3d26073f9acf, args=TestStorageParam(hostIdList=null, clusterId=2, obBackupStorageBaseInfo=ObBackupStorageBaseInfo(backupStorageType=BACKUP_STORAGE_S3, storageUrl=kungeek-oceanbase-******/test_all_ob_cluster_inventory/test/backupsets, ossAccessKey=null, cosAccessKey=null, obsAccessKey=null, s3AccessKey=S3AccessKey(region=ap-beijing, endpoint=cos.ap-beijing.myqcloud.com, accessKeyId=xxx), customBackupRangeList=[], advancedConfigList=[BackupAdvancedConfigInfo(name=delete_mode, values=[delete]), BackupAdvancedConfigInfo(name=checksum_type, values=[md5]), BackupAdvancedConfigInfo(name=addressing_model, values=[virtual_hosted_style])]), backupPurposeType=BACKUP, storageId=null),, duration=172 ms] com.oceanbase.ocp.core.exception.UnexpectedException: [OCP UnexpectedException]: status=500 INTERNAL_SERVER_ERROR, errorCode=BACKUP_CONFIG_STORAGE_TEST_FAIL, args=hostIpList:[172.18.65.23, 172.18.65.35, 172.18.64.87],s3://kungeek-oceanbase-******/test_all_ob_cluster_inventory/test/backupsets at com.oceanbase.ocp.backup.internal.utils.ObBackupChecker.checkStorageAccessible(ObBackupChecker.java:478) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl.testStorageConfig(BackupCommonServiceImpl.java:648) at jdk.internal.reflect.GeneratedMethodAccessor3990.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:724) at com.oceanbase.ocp.backup.internal.facadeimpl.common.BackupCommonServiceImpl$$SpringCGLIB$$0.testStorageConfig() at com.oceanbase.ocp.server.common.controller.ObBackupController.testStorageConfig(ObBackupController.java:227) at jdk.internal.reflect.GeneratedMethodAccessor3469.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:132) at jdk.internal.reflect.GeneratedMethodAccessor447.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:642) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:632) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:71) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.proceed(AuthorizationManagerBeforeMethodInterceptor.java:268) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.attemptAuthorization(AuthorizationManagerBeforeMethodInterceptor.java:263) at org.springframework.security.authorization.method.AuthorizationManagerBeforeMethodInterceptor.invoke(AuthorizationManagerBeforeMethodInterceptor.java:196) at org.springframework.security.config.annotation.method.configuration.DeferringMethodInterceptor.invoke(DeferringMethodInterceptor.java:44) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:728) at com.oceanbase.ocp.server.common.controller.ObBackupController$$SpringCGLIB$$0.testStorageConfig() at jdk.internal.reflect.GeneratedMethodAccessor3469.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231) at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340) at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128) at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:101) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:50) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.saml2.provider.service.web.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:198) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:131) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$4(HandlerMappingIntrospector.java:267) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113) at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:238) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:362) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:278) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:116) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:398) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:903) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1769) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1189) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:658) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang.Thread.run(Unknown Source) 2025-12-19 16:08:41.997 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-5,92325726fdc95a59,acd16229c19d4dc5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=92325726fdc95a59, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:08:41.997 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-5,92325726fdc95a59,acd16229c19d4dc5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=92325726fdc95a59, duration=1 ms] 2025-12-19 16:08:42.652 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-1,b250e03ebd4a28fc,22803dde742ca362] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.34, traceId=b250e03ebd4a28fc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:08:42.652 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-1,b250e03ebd4a28fc,22803dde742ca362] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.34, traceId=b250e03ebd4a28fc, duration=0 ms] 2025-12-19 16:08:43.202 WARN 3588644 --- [ocp-async-5,91d94415ef11b499,591e660085be992f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:08:43.210 INFO 3588644 --- [prometheus-minute-schedule-1,b4dc7e0e750c0283,ee221454c8748938] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1766131740, interval=60 2025-12-19 16:08:43.214 INFO 3588644 --- [prometheus-minute-schedule-1,b4dc7e0e750c0283,ee221454c8748938] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-12-19 16:08:43.419 INFO 3588644 --- [ocp-async-1,2fddd138a865e984,842fd55034de558f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:08:43.445 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-10,cfc5e73a7d6815d4,d2a72f7af0bbec23] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=test_all_ob_cluster, client=172.21.96.40, traceId=cfc5e73a7d6815d4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=test_all_ob_cluster,false,0,] 2025-12-19 16:08:43.445 INFO 3588644 --- [http-nio-0.0.0.0-8080-exec-10,cfc5e73a7d6815d4,d2a72f7af0bbec23] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=172.21.96.40, traceId=cfc5e73a7d6815d4, duration=0 ms] 2025-12-19 16:08:43.613 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-12-19T16:08:43.613664485+08:00, tenantSize=8 2025-12-19 16:08:43.616 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5currentTimeUs:1766131723613697 2025-12-19 16:08:43.616 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5diagnosisStartTimeUs:1766131423580207 2025-12-19 16:08:43.621 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_5totalLength:0 2025-12-19 16:08:43.626 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_8currentTimeUs:1766131723624972 2025-12-19 16:08:43.626 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_8diagnosisStartTimeUs:1766131423601450 2025-12-19 16:08:43.632 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_8totalLength:0 2025-12-19 16:08:43.638 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_7currentTimeUs:1766131723637005 2025-12-19 16:08:43.638 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_7diagnosisStartTimeUs:1766131423591249 2025-12-19 16:08:43.644 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_7totalLength:0 2025-12-19 16:08:43.648 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1766131723647104 2025-12-19 16:08:43.648 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1766131423545597 2025-12-19 16:08:43.653 INFO 3588644 --- [ocp-schedules-25,07f948e6a74f64f7,af71737a2d032afe] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-12-19 16:08:43.918 INFO 3588644 --- [ocp-schedules-21,ea668010eee140a8,873d3db9d6dd64b9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-12-19 16:08:43.919 INFO 3588644 --- [ocp-schedules-21,ea668010eee140a8,873d3db9d6dd64b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-12-19 16:08:43.924 INFO 3588644 --- [ocp-schedules-18,c21fdc76aa73237c,1d6c4e99ab44bfe2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-12-19 16:08:43.925 INFO 3588644 --- [ocp-schedules-18,c21fdc76aa73237c,1d6c4e99ab44bfe2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-12-19 16:08:43.925 INFO 3588644 --- [ocp-schedules-18,c21fdc76aa73237c,1d6c4e99ab44bfe2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-12-19 16:08:43.927 INFO 3588644 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:43.927 INFO 3588644 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-12-19 16:08:43.928 INFO 3588644 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.8 2025-12-19 16:08:43.931 INFO 3588644 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:43.933 INFO 3588644 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-12-19 16:08:43.938 INFO 3588644 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.4.1.0 2025-12-19 16:08:43.938 INFO 3588644 --- [ocp-schedules-21,ea668010eee140a8,873d3db9d6dd64b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-12-19 16:08:43.945 INFO 3588644 --- [ocp-schedules-10,a47d9051b3622819,5cecc8e5ad0c321c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-12-19 16:08:43.959 INFO 3588644 --- [ocp-schedules-29,e08c4f8b3b57fa84,50eb8a5e7b36edae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-12-19 16:08:43.960 INFO 3588644 --- [ocp-schedules-29,e08c4f8b3b57fa84,50eb8a5e7b36edae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-12-19 16:08:43.961 INFO 3588644 --- [ocp-schedules-7,3be2faf6742cdc96,991005a929266f3f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-12-19 16:08:43.963 INFO 3588644 --- [ocp-schedules-16,f9d8768b1b8fa22c,252834b6974e62b3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[172.18.96.94, 172.18.65.23, 172.18.65.35, 172.18.64.87, 172.18.64.252] 2025-12-19 16:08:43.963 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=172.18.96.94, originalStatus=ONLINE 2025-12-19 16:08:43.963 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=172.18.65.23, originalStatus=ONLINE 2025-12-19 16:08:43.963 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=172.18.65.35, originalStatus=ONLINE 2025-12-19 16:08:43.964 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=172.18.64.87, originalStatus=ONLINE 2025-12-19 16:08:43.964 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=6, ip=172.18.64.252, originalStatus=ONLINE 2025-12-19 16:08:43.964 INFO 3588644 --- [ocp-schedules-5,560fe0d5750b5c39,1603b7f51aeaa6d0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[172.18.96.94, 172.18.65.23, 172.18.65.35, 172.18.64.87, 172.18.64.252] 2025-12-19 16:08:43.965 INFO 3588644 --- [host-check-clockdiff1,560fe0d5750b5c39,a9d13049f37d6294] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.18.96.94], timeoutMillis=20000 2025-12-19 16:08:43.965 INFO 3588644 --- [host-check-clockdiff2,560fe0d5750b5c39,00018223a05191ef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.18.65.23], timeoutMillis=20000 2025-12-19 16:08:43.966 INFO 3588644 --- [ocp-schedules-23,32ce17e13b9c2784,188dcaa5e73fe5dd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[172.18.96.94, 172.18.65.23, 172.18.65.35, 172.18.64.87, 172.18.64.252] 2025-12-19 16:08:43.966 INFO 3588644 --- [host-check-clockdiff3,560fe0d5750b5c39,5642a5c0ed220ae6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.18.65.35], timeoutMillis=20000 2025-12-19 16:08:43.966 INFO 3588644 --- [host-check-clockdiff4,560fe0d5750b5c39,0604bc8f03412286] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.18.64.87], timeoutMillis=20000 2025-12-19 16:08:43.967 INFO 3588644 --- [host-check-agent-status1,32ce17e13b9c2784,3335f6c3010c4ef1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=172.18.96.94, originalStatus=ONLINE 2025-12-19 16:08:43.967 INFO 3588644 --- [host-check-clockdiff5,560fe0d5750b5c39,cf495d832dc5bae3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.18.64.252], timeoutMillis=20000 2025-12-19 16:08:43.967 INFO 3588644 --- [host-check-agent-status2,32ce17e13b9c2784,ed5aedd5c1f9ad5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=172.18.65.23, originalStatus=ONLINE 2025-12-19 16:08:43.968 INFO 3588644 --- [host-check-agent-status3,32ce17e13b9c2784,bed6216615d77e80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=172.18.65.35, originalStatus=ONLINE 2025-12-19 16:08:43.969 INFO 3588644 --- [host-check-agent-status4,32ce17e13b9c2784,66cfbe9d62578fae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=172.18.64.87, originalStatus=ONLINE 2025-12-19 16:08:43.970 INFO 3588644 --- [host-check-agent-status5,32ce17e13b9c2784,05e089c60b14c422] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=6, ip=172.18.64.252, originalStatus=ONLINE 2025-12-19 16:08:43.971 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-12-19 16:08:43.974 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-12-19 16:08:43.974 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-12-19 16:08:43.977 INFO 3588644 --- [host-check-agent-status4,32ce17e13b9c2784,66cfbe9d62578fae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-12-19 16:08:43.977 INFO 3588644 --- [host-check-agent-status2,32ce17e13b9c2784,ed5aedd5c1f9ad5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-12-19 16:08:43.977 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-12-19 16:08:43.977 INFO 3588644 --- [ocp-schedules-17,39ff52a46a464a9e,da9ba07825d2a9d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-12-19 16:08:43.978 INFO 3588644 --- [host-check-agent-status3,32ce17e13b9c2784,bed6216615d77e80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-12-19 16:08:43.981 INFO 3588644 --- [host-check-agent-status2,32ce17e13b9c2784,ed5aedd5c1f9ad5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-12-19 16:08:43.982 INFO 3588644 --- [host-check-clockdiff1,560fe0d5750b5c39,a9d13049f37d6294] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1766131723 0 0 , error= 2025-12-19 16:08:43.984 INFO 3588644 --- [host-check-agent-status4,32ce17e13b9c2784,66cfbe9d62578fae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-12-19 16:08:43.985 INFO 3588644 --- [host-check-agent-status5,32ce17e13b9c2784,05e089c60b14c422] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=6 2025-12-19 16:08:43.987 INFO 3588644 --- [host-check-agent-status5,32ce17e13b9c2784,05e089c60b14c422] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2025-12-19 16:08:43.989 INFO 3588644 --- [ocp-schedules-24,0efcccabcc58341b,a87640448ab327c9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-12-19 16:08:43.989 INFO 3588644 --- [host-check-agent-status3,32ce17e13b9c2784,bed6216615d77e80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-12-19 16:08:43.990 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-12-19 16:08:43.993 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=6 2025-12-19 16:08:43.994 INFO 3588644 --- [alarm-task-1,7e7b76990905e0c5,8ad399334d48fd6d] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-12-19T16:08:43.992941+08:00, lastProcessTime=2025-12-19T16:08:33.985745+08:00 2025-12-19 16:08:43.995 INFO 3588644 --- [ocp-schedules-17,39ff52a46a464a9e,da9ba07825d2a9d4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-12-19 16:08:43.997 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-12-19 16:08:43.999 INFO 3588644 --- [host-check-agent-status1,32ce17e13b9c2784,3335f6c3010c4ef1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-12-19 16:08:44.002 INFO 3588644 --- [ocp-schedules-24,0efcccabcc58341b,a87640448ab327c9] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-12-19 16:08:44.003 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-12-19 16:08:44.003 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-12-19 16:08:44.005 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.64.87:62888/api/v1/time, params:null 2025-12-19 16:08:44.006 INFO 3588644 --- [host-check-agent-status5,32ce17e13b9c2784,05e089c60b14c422] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.64.252:62888/api/v1/agent/status, params:null 2025-12-19 16:08:44.007 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.65.23:62888/api/v1/time, params:null 2025-12-19 16:08:44.007 INFO 3588644 --- [host-check-agent-status2,32ce17e13b9c2784,ed5aedd5c1f9ad5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.65.23:62888/api/v1/agent/status, params:null 2025-12-19 16:08:44.008 INFO 3588644 --- [host-check-agent-status1,32ce17e13b9c2784,3335f6c3010c4ef1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-12-19 16:08:44.009 INFO 3588644 --- [host-check-agent-status4,32ce17e13b9c2784,66cfbe9d62578fae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.64.87:62888/api/v1/agent/status, params:null 2025-12-19 16:08:44.010 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=172.18.64.87, timestamp=2025-12-19T16:08:44.009066+08:00 2025-12-19 16:08:44.010 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-12-19T16:08:44.010427648+08:00 2025-12-19 16:08:44.010 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=172.18.65.23, timestamp=2025-12-19T16:08:44.009339401+08:00 2025-12-19 16:08:44.011 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-12-19T16:08:44.010844742+08:00 2025-12-19 16:08:44.012 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2025-12-19 16:08:44.012 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-12-19T16:08:44.010427648+08:00 and status AVAILABLE 2025-12-19 16:08:44.013 INFO 3588644 --- [host-check-agent-status2,32ce17e13b9c2784,ed5aedd5c1f9ad5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.18.65.23 2025-12-19 16:08:44.013 INFO 3588644 --- [host-check-agent-status2,32ce17e13b9c2784,ed5aedd5c1f9ad5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.18.65.23 2025-12-19 16:08:44.013 INFO 3588644 --- [host-check-agent-status2,32ce17e13b9c2784,ed5aedd5c1f9ad5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.18.65.23 2025-12-19 16:08:44.013 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-12-19T16:08:44.010844742+08:00 and status AVAILABLE 2025-12-19 16:08:44.014 INFO 3588644 --- [host-check-agent-status5,32ce17e13b9c2784,05e089c60b14c422] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.18.64.252 2025-12-19 16:08:44.014 INFO 3588644 --- [host-check-agent-status5,32ce17e13b9c2784,05e089c60b14c422] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.18.64.252 2025-12-19 16:08:44.014 INFO 3588644 --- [host-check-agent-status5,32ce17e13b9c2784,05e089c60b14c422] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.18.64.252 2025-12-19 16:08:44.014 INFO 3588644 --- [host-check-agent-status3,32ce17e13b9c2784,bed6216615d77e80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.65.35:62888/api/v1/agent/status, params:null 2025-12-19 16:08:44.016 INFO 3588644 --- [host-check-agent-status4,32ce17e13b9c2784,66cfbe9d62578fae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.18.64.87 2025-12-19 16:08:44.016 INFO 3588644 --- [host-check-agent-status4,32ce17e13b9c2784,66cfbe9d62578fae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.18.64.87 2025-12-19 16:08:44.016 INFO 3588644 --- [host-check-agent-status4,32ce17e13b9c2784,66cfbe9d62578fae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.18.64.87 2025-12-19 16:08:44.017 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-12-19 16:08:44.018 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.018 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-12-19 16:08:44.018 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-12-19 16:08:44.020 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.021 INFO 3588644 --- [host-check-agent-status3,32ce17e13b9c2784,bed6216615d77e80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.18.65.35 2025-12-19 16:08:44.021 INFO 3588644 --- [host-check-agent-status3,32ce17e13b9c2784,bed6216615d77e80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.18.65.35 2025-12-19 16:08:44.021 INFO 3588644 --- [host-check-agent-status3,32ce17e13b9c2784,bed6216615d77e80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.18.65.35 2025-12-19 16:08:44.022 INFO 3588644 --- [host-check-agent-status1,32ce17e13b9c2784,3335f6c3010c4ef1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.96.94:62888/api/v1/agent/status, params:null 2025-12-19 16:08:44.022 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.65.35:62888/api/v1/time, params:null 2025-12-19 16:08:44.025 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.96.94:62888/api/v1/time, params:null 2025-12-19 16:08:44.025 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-12-19 16:08:44.026 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=172.18.65.35, timestamp=2025-12-19T16:08:44.023997140+08:00 2025-12-19 16:08:44.027 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=172.18.96.94, timestamp=2025-12-19T16:08:44.026956197+08:00 2025-12-19 16:08:44.027 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-12-19 16:08:44.028 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-12-19T16:08:44.027079709+08:00 2025-12-19 16:08:44.028 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-12-19T16:08:44.027402626+08:00 2025-12-19 16:08:44.029 INFO 3588644 --- [host-check-agent-status1,32ce17e13b9c2784,3335f6c3010c4ef1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.18.96.94 2025-12-19 16:08:44.029 INFO 3588644 --- [host-check-agent-status1,32ce17e13b9c2784,3335f6c3010c4ef1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.18.96.94 2025-12-19 16:08:44.029 INFO 3588644 --- [host-check-agent-status1,32ce17e13b9c2784,3335f6c3010c4ef1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.18.96.94 2025-12-19 16:08:44.030 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-12-19 16:08:44.030 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-12-19T16:08:44.027402626+08:00 and status AVAILABLE 2025-12-19 16:08:44.030 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-12-19T16:08:44.027079709+08:00 and status AVAILABLE 2025-12-19 16:08:44.037 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-12-19 16:08:44.039 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-12-19 16:08:44.041 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.23:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-12-19 16:08:44.041 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-12-19 16:08:44.042 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-12-19 16:08:44.043 INFO 3588644 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test_ocp_ob_cluster, tenantName=sys, dbUser=ocp_monitor 2025-12-19 16:08:44.044 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.044 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-12-19 16:08:44.048 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.18.64.252:62888/api/v1/time, params:null 2025-12-19 16:08:44.050 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=6, ip=172.18.64.252, timestamp=2025-12-19T16:08:44.049458021+08:00 2025-12-19 16:08:44.051 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=6, lastAvailableTime=2025-12-19T16:08:44.050850020+08:00 2025-12-19 16:08:44.052 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-12-19 16:08:44.052 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-12-19 16:08:44.052 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.s.compute.HostAgentDaoManager : Updated agent 6 last available time 2025-12-19T16:08:44.050850020+08:00 and status AVAILABLE 2025-12-19 16:08:44.053 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.64.87:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-12-19 16:08:44.055 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-12-19 16:08:44.055 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-12-19 16:08:44.056 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=6 2025-12-19 16:08:44.064 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=6 2025-12-19 16:08:44.067 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 6 2025-12-19 16:08:44.067 INFO 3588644 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test_ocp_ob_cluster, tenantName=sys, dbUser=ocp_monitor 2025-12-19 16:08:44.068 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.068 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-12-19 16:08:44.069 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.35:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-12-19 16:08:44.069 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.96.94:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-12-19 16:08:44.069 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.074 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-12-19 16:08:44.076 INFO 3588644 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-12-19 16:08:44.081 INFO 3588644 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-12-19 16:08:44.081 INFO 3588644 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=172.18.96.94, port=2882, sqlPort=2881, installPath=/data/oceanbase, runPath=/data/oceanbase, runUser=root, version=4.2.1.8_108000022024072217-3149c25ca2dadbb7707686ad02a1367b1b43e0b5(Jul 23 2024 02:01:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-27T18:02:41+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.18.96.94, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.8_108000022024072217-3149c25ca2dadbb7707686ad02a1367b1b43e0b5(Jul 23 2024 02:01:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-27T18:02:40.761447+08:00, stopTime=1970-01-01T08:00+08:00} 2025-12-19 16:08:44.085 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.64.252:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-12-19 16:08:44.090 INFO 3588644 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test_all_ob_cluster, tenantName=sys, dbUser=ocp_monitor 2025-12-19 16:08:44.094 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.096 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-12-19 16:08:44.100 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=28 2025-12-19 16:08:44.101 INFO 3588644 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=172.18.96.94, port=2882, sqlPort=2881, installPath=/data/oceanbase, runPath=/data/oceanbase, runUser=root, version=4.2.1.8_108000022024072217-3149c25ca2dadbb7707686ad02a1367b1b43e0b5(Jul 23 2024 02:01:58), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-27T18:02:40.761447+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-12-19 16:08:44.105 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=25 2025-12-19 16:08:44.106 INFO 3588644 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test_ocp_ob_cluster, tenantName=sys, dbUser=ocp_monitor 2025-12-19 16:08:44.106 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=28 2025-12-19 16:08:44.106 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.107 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-12-19 16:08:44.108 INFO 3588644 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster test_ocp_ob_cluster 2025-12-19 16:08:44.109 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=19 2025-12-19 16:08:44.112 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=25 2025-12-19 16:08:44.112 INFO 3588644 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-12-19 16:08:44.117 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=19 2025-12-19 16:08:44.119 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.23:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-12-19 16:08:44.119 INFO 3588644 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-12-19 16:08:44.119 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.96.94:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-12-19 16:08:44.119 INFO 3588644 --- [alarm-task-1,7e7b76990905e0c5,8ad399334d48fd6d] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-12-19T16:08:43.992941+08:00 2025-12-19 16:08:44.121 INFO 3588644 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-12-19 16:08:44.125 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.64.87:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-12-19 16:08:44.126 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=22 2025-12-19 16:08:44.127 INFO 3588644 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test_all_ob_cluster, tenantName=sys, dbUser=ocp_monitor 2025-12-19 16:08:44.131 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.133 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-12-19 16:08:44.133 INFO 3588644 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.96.94:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-12-19 16:08:44.135 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=22 2025-12-19 16:08:44.137 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.142 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-12-19 16:08:44.142 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.35:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-12-19 16:08:44.144 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=31 2025-12-19 16:08:44.144 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-12-19 16:08:44.151 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-12-19 16:08:44.151 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=31 2025-12-19 16:08:44.151 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=172.18.64.87, port=2882, sqlPort=2881, installPath=/data/oceanbase, runPath=/data/oceanbase, runUser=admin, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:43:04+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.18.64.87, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:43:03.943169+08:00, stopTime=1970-01-01T08:00+08:00} 2025-12-19 16:08:44.155 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=29 2025-12-19 16:08:44.161 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=29 2025-12-19 16:08:44.164 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=172.18.64.87, port=2882, sqlPort=2881, installPath=/data/oceanbase, runPath=/data/oceanbase, runUser=admin, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=false, hostId=4, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:43:03.943169+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-12-19 16:08:44.164 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-12-19 16:08:44.166 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.64.252:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-12-19 16:08:44.168 INFO 3588644 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=82189, runPath=/data/oceanbase, runUser=root, execute=/data/oceanbase/bin/observer, installPath=/data/oceanbase) 2025-12-19 16:08:44.168 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.169 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.96.94:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-12-19 16:08:44.169 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-12-19 16:08:44.170 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-12-19 16:08:44.170 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=172.18.65.35, port=2882, sqlPort=2881, installPath=/data/oceanbase, runPath=/data/oceanbase, runUser=admin, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:41:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.18.65.35, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:41:06.447376+08:00, stopTime=1970-01-01T08:00+08:00} 2025-12-19 16:08:44.177 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=20 2025-12-19 16:08:44.181 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=172.18.65.35, port=2882, sqlPort=2881, installPath=/data/oceanbase, runPath=/data/oceanbase, runUser=admin, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:41:06.447376+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-12-19 16:08:44.181 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-12-19 16:08:44.183 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=26 2025-12-19 16:08:44.183 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=20 2025-12-19 16:08:44.187 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-12-19 16:08:44.187 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=172.18.65.23, port=2882, sqlPort=2881, installPath=/data/oceanbase, runPath=/data/oceanbase, runUser=admin, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:39:12+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.18.65.23, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:39:11.976258+08:00, stopTime=1970-01-01T08:00+08:00} 2025-12-19 16:08:44.189 WARN 3588644 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-12-19 16:08:44.190 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=26 2025-12-19 16:08:44.190 INFO 3588644 --- [pool-13-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 172.18.96.94:8080 refreshes policies... 2025-12-19 16:08:44.190 INFO 3588644 --- [pool-13-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-12-19 16:08:44.194 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.64.87:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-12-19 16:08:44.195 INFO 3588644 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test_ocp_ob_cluster, tenantName=sys, dbUser=ocp_monitor 2025-12-19 16:08:44.196 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.196 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=30 2025-12-19 16:08:44.196 INFO 3588644 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-12-19 16:08:44.197 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=172.18.65.23, port=2882, sqlPort=2881, installPath=/data/oceanbase, runPath=/data/oceanbase, runUser=admin, version=4.4.1.0_100000032025101610-70824e90816ed333c56f3dd1af34ffec43d0fa88(Oct 16 2025 10:13:49), withRootserver=true, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-12-15T10:39:11.976258+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-12-19 16:08:44.198 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.23:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-12-19 16:08:44.199 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=23 2025-12-19 16:08:44.202 INFO 3588644 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=test_ocp_ob_cluster, obClusterId=1761559195 2025-12-19 16:08:44.202 INFO 3588644 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=test_ocp_ob_cluster, obClusterId=1761559195, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"test_ocp_ob_cluster","ObClusterId":1761559195,"ObRegion":"test_ocp_ob_cluster","ObRegionId":1761559195,"ReadonlyRsList":[],"RsList":[{"address":"172.18.96.94:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1766131363480000}, createTime=2025-10-27T18:10:09+08:00, creator=admin, obVersion=4.2.1.8, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/data/oceanbase/test_ocp_ob_cluster/datafile","DiskPathStyle":"OBD","InstallPath":"/data/oceanbase","LogDiskPath":"/data/oceanbase","OperatingSystemUser":"root","RunPath":"/data/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=3, clusterMode=DISTRIBUTED_SHARED_NOTHING) 2025-12-19 16:08:44.204 INFO 3588644 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test_all_ob_cluster, tenantName=sys, dbUser=ocp_monitor 2025-12-19 16:08:44.204 INFO 3588644 --- [host-check-host-status1,f9d8768b1b8fa22c,19ff88c625bbb0f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=30 2025-12-19 16:08:44.206 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=23 2025-12-19 16:08:44.207 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.207 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=32 2025-12-19 16:08:44.209 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-12-19 16:08:44.213 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=32 2025-12-19 16:08:44.214 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.35:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-12-19 16:08:44.214 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster test_all_ob_cluster 2025-12-19 16:08:44.216 INFO 3588644 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-12-19 16:08:44.219 INFO 3588644 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-12-19 16:08:44.221 INFO 3588644 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-12-19 16:08:44.222 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.64.252:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-12-19 16:08:44.230 INFO 3588644 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.18.65.23:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-12-19 16:08:44.242 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=21 2025-12-19 16:08:44.248 INFO 3588644 --- [host-check-host-status4,f9d8768b1b8fa22c,9850fb01b21c2fd4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=21 2025-12-19 16:08:44.258 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=27 2025-12-19 16:08:44.263 INFO 3588644 --- [host-check-host-status2,f9d8768b1b8fa22c,64180ab17c56fbcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=27 2025-12-19 16:08:44.267 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=24 2025-12-19 16:08:44.269 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=33 2025-12-19 16:08:44.274 INFO 3588644 --- [host-check-host-status3,f9d8768b1b8fa22c,2fecde23a6aa79db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=24 2025-12-19 16:08:44.275 INFO 3588644 --- [host-check-host-status5,f9d8768b1b8fa22c,d6472177cfb7eeee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=33 2025-12-19 16:08:44.395 INFO 3588644 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=3197823, runPath=/data/oceanbase, runUser=admin, execute=/data/oceanbase/bin/observer, installPath=/data/oceanbase) 2025-12-19 16:08:44.397 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.399 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-12-19 16:08:44.403 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.406 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-12-19 16:08:44.412 WARN 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-12-19 16:08:44.417 INFO 3588644 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=test_all_ob_cluster, tenantName=sys, dbUser=ocp_monitor 2025-12-19 16:08:44.420 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-12-19 16:08:44.423 INFO 3588644 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-12-19 16:08:44.430 INFO 3588644 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=test_all_ob_cluster, obClusterId=1761559196 2025-12-19 16:08:44.430 INFO 3588644 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=test_all_ob_cluster, obClusterId=1761559196, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"test_all_ob_cluster","ObClusterId":1761559196,"ObRegion":"test_all_ob_cluster","ObRegionId":1761559196,"ReadonlyRsList":[],"RsList":[{"address":"172.18.65.23:2882","role":"LEADER","sql_port":2881},{"address":"172.18.64.87:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.18.65.35:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1766131158874000}, createTime=2025-10-27T18:56:29+08:00, creator=admin, obVersion=4.4.1.0, packageOperatingSystem=el8, loadType=HTAP, attributesJson={"DataDiskPath":"/data/oceanbase/test_all_ob_cluster/datafile","DiskPathStyle":"DEFAULT","InstallPath":"/data/oceanbase","LogDiskPath":"/data/oceanbase/test_all_ob_cluster/clog","OperatingSystemUser":"admin","RunPath":"/data/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=5, clusterMode=DISTRIBUTED_SHARED_NOTHING) 2025-12-19 16:08:44.430 INFO 3588644 --- [ocp-schedules-29,e08c4f8b3b57fa84,50eb8a5e7b36edae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-12-19 16:08:45.102 INFO 3588644 --- [prometheus-second-schedule-1,b6b130a4b7cedc13,d9440aef49ee353c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-12-19 16:08:45.202 WARN 3588644 --- [ocp-async-9,e903a5d97d97e8bb,e4f897672eec80a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-12-19 16:08:45.318 INFO 3588644 --- [ocp-schedules-24,66ae298ef575976e,90238ee6be8e2c2d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-12-19 16:08:45.398 INFO 3588644 --- [ocp-schedules-23,f34ea35b2f508ff3,cb4081bf7ba22004] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-12-19 16:08:45.419 INFO 3588644 --- [ocp-async-2,1578b9f50e1941c1,6649241f7ef24151] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-12-19 16:08:45.853 INFO 3588644 --- [ocp-schedules-20,f0c97f5cb71eab31,8db73652c49bb416] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start ^C