[2024-10-16 09:39:22.509] [DEBUG] - cmd: ['cluster1'] [2024-10-16 09:39:22.509] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-10-16 09:39:22.510] [DEBUG] - mkdir /root/.obd/lock/ [2024-10-16 09:39:22.510] [DEBUG] - unknown lock mode [2024-10-16 09:39:22.510] [DEBUG] - try to get share lock /root/.obd/lock/global [2024-10-16 09:39:22.510] [DEBUG] - share lock `/root/.obd/lock/global`, count 1 [2024-10-16 09:39:22.510] [DEBUG] - Get Deploy by name [2024-10-16 09:39:22.510] [DEBUG] - mkdir /root/.obd/cluster/ [2024-10-16 09:39:22.511] [DEBUG] - mkdir /root/.obd/config_parser/ [2024-10-16 09:39:22.511] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_cluster1 [2024-10-16 09:39:22.511] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_cluster1`, count 1 [2024-10-16 09:39:22.520] [DEBUG] - Deploy status judge [2024-10-16 09:39:22.520] [INFO] Get local repositories [2024-10-16 09:39:22.521] [DEBUG] - mkdir /root/.obd/repository [2024-10-16 09:39:22.521] [DEBUG] - Get local repository oceanbase-ce-4.2.4.0-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.522] [DEBUG] - Search repository oceanbase-ce version: 4.2.4.0, tag: 8b1365b55251aae29758571ac6009989f2262ca9, release: None, package_hash: None [2024-10-16 09:39:22.522] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo [2024-10-16 09:39:22.522] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 1 [2024-10-16 09:39:22.522] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce [2024-10-16 09:39:22.525] [DEBUG] - Found repository oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.525] [DEBUG] - Get local repository obproxy-ce-4.3.1.0-None [2024-10-16 09:39:22.525] [DEBUG] - Search repository obproxy-ce version: 4.3.1.0, tag: None, release: None, package_hash: None [2024-10-16 09:39:22.525] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 2 [2024-10-16 09:39:22.525] [DEBUG] - mkdir /root/.obd/repository/obproxy-ce [2024-10-16 09:39:22.531] [DEBUG] - Found repository obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:22.531] [DEBUG] - Get local repository obagent-4.2.2-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.531] [DEBUG] - Search repository obagent version: 4.2.2, tag: d8172a2bf3811219777124490558bf800dffef87, release: None, package_hash: None [2024-10-16 09:39:22.531] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 3 [2024-10-16 09:39:22.532] [DEBUG] - mkdir /root/.obd/repository/obagent [2024-10-16 09:39:22.534] [DEBUG] - Found repository obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.534] [DEBUG] - Get local repository ocp-express-4.2.2-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.535] [DEBUG] - Search repository ocp-express version: 4.2.2, tag: 44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e, release: None, package_hash: None [2024-10-16 09:39:22.535] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 4 [2024-10-16 09:39:22.535] [DEBUG] - mkdir /root/.obd/repository/ocp-express [2024-10-16 09:39:22.538] [DEBUG] - Found repository ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.652] [DEBUG] - Get deploy config [2024-10-16 09:39:22.684] [INFO] Search plugins [2024-10-16 09:39:22.684] [DEBUG] - Searching start_check plugin for components ... [2024-10-16 09:39:22.684] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.685] [DEBUG] - mkdir /root/.obd/plugins [2024-10-16 09:39:22.686] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.4.0 [2024-10-16 09:39:22.686] [DEBUG] - Searching start_check plugin for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:22.686] [DEBUG] - Found for obproxy-ce-py_script_start_check-4.2.3 for obproxy-ce-4.3.1.0 [2024-10-16 09:39:22.686] [DEBUG] - Searching start_check plugin for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.687] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-4.2.2 [2024-10-16 09:39:22.687] [DEBUG] - Searching start_check plugin for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.687] [DEBUG] - Found for ocp-express-py_script_start_check-4.2.2 for ocp-express-4.2.2 [2024-10-16 09:39:22.687] [DEBUG] - Searching create_tenant plugin for components ... [2024-10-16 09:39:22.687] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.688] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.4.0 [2024-10-16 09:39:22.688] [DEBUG] - Searching create_tenant plugin for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:22.689] [DEBUG] - No such create_tenant plugin for obproxy-ce-4.3.1.0 [2024-10-16 09:39:22.689] [DEBUG] - Searching create_tenant plugin for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.689] [DEBUG] - No such create_tenant plugin for obagent-4.2.2 [2024-10-16 09:39:22.689] [DEBUG] - Searching create_tenant plugin for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.689] [DEBUG] - No such create_tenant plugin for ocp-express-4.2.2 [2024-10-16 09:39:22.689] [DEBUG] - Searching tenant_optimize plugin for components ... [2024-10-16 09:39:22.689] [DEBUG] - Searching tenant_optimize plugin for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.690] [DEBUG] - No such tenant_optimize plugin for oceanbase-ce-4.2.4.0 [2024-10-16 09:39:22.690] [DEBUG] - Searching tenant_optimize plugin for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:22.690] [DEBUG] - No such tenant_optimize plugin for obproxy-ce-4.3.1.0 [2024-10-16 09:39:22.690] [DEBUG] - Searching tenant_optimize plugin for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.690] [DEBUG] - No such tenant_optimize plugin for obagent-4.2.2 [2024-10-16 09:39:22.691] [DEBUG] - Searching tenant_optimize plugin for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.691] [DEBUG] - No such tenant_optimize plugin for ocp-express-4.2.2 [2024-10-16 09:39:22.691] [DEBUG] - Searching start plugin for components ... [2024-10-16 09:39:22.691] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.692] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.4.0 [2024-10-16 09:39:22.692] [DEBUG] - Searching start plugin for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:22.692] [DEBUG] - Found for obproxy-ce-py_script_start-4.2.3 for obproxy-ce-4.3.1.0 [2024-10-16 09:39:22.692] [DEBUG] - Searching start plugin for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.693] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-4.2.2 [2024-10-16 09:39:22.693] [DEBUG] - Searching start plugin for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.693] [DEBUG] - Found for ocp-express-py_script_start-4.2.2 for ocp-express-4.2.2 [2024-10-16 09:39:22.693] [DEBUG] - Searching connect plugin for components ... [2024-10-16 09:39:22.693] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.694] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.2.0 for oceanbase-ce-4.2.4.0 [2024-10-16 09:39:22.694] [DEBUG] - Searching connect plugin for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:22.694] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.3.1.0 [2024-10-16 09:39:22.694] [DEBUG] - Searching connect plugin for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.695] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-4.2.2 [2024-10-16 09:39:22.695] [DEBUG] - Searching connect plugin for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.695] [DEBUG] - Found for ocp-express-py_script_connect-1.0.1 for ocp-express-4.2.2 [2024-10-16 09:39:22.695] [DEBUG] - Searching bootstrap plugin for components ... [2024-10-16 09:39:22.695] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.696] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.2.0 for oceanbase-ce-4.2.4.0 [2024-10-16 09:39:22.696] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:22.696] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.3.1.0 [2024-10-16 09:39:22.696] [DEBUG] - Searching bootstrap plugin for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.697] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-4.2.2 [2024-10-16 09:39:22.697] [DEBUG] - Searching bootstrap plugin for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.697] [DEBUG] - Found for ocp-express-py_script_bootstrap-1.0.1 for ocp-express-4.2.2 [2024-10-16 09:39:22.697] [DEBUG] - Searching display plugin for components ... [2024-10-16 09:39:22.697] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.698] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.4.0 [2024-10-16 09:39:22.698] [DEBUG] - Searching display plugin for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:22.698] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.3.1.0 [2024-10-16 09:39:22.698] [DEBUG] - Searching display plugin for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.699] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-4.2.2 [2024-10-16 09:39:22.699] [DEBUG] - Searching display plugin for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.699] [DEBUG] - Found for ocp-express-py_script_display-1.0.1 for ocp-express-4.2.2 [2024-10-16 09:39:22.815] [INFO] Load cluster param plugin [2024-10-16 09:39:22.816] [DEBUG] - Get local repository oceanbase-ce-4.2.4.0-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:22.816] [DEBUG] - Get local repository obproxy-ce-4.3.1.0-None [2024-10-16 09:39:22.816] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 5 [2024-10-16 09:39:22.819] [DEBUG] - Get local repository obagent-4.2.2-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:22.819] [DEBUG] - Get local repository ocp-express-4.2.2-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:22.820] [DEBUG] - Searching param plugin for components ... [2024-10-16 09:39:22.820] [DEBUG] - Search param plugin for oceanbase-ce [2024-10-16 09:39:22.820] [DEBUG] - Found for oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.4.0 [2024-10-16 09:39:22.820] [DEBUG] - Applying oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:23.589] [DEBUG] - Search param plugin for obproxy-ce [2024-10-16 09:39:23.590] [DEBUG] - Found for obproxy-ce-param-4.2.3 for obproxy-ce-4.3.1.0 [2024-10-16 09:39:23.590] [DEBUG] - Applying obproxy-ce-param-4.2.3 for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:23.769] [DEBUG] - Search param plugin for obagent [2024-10-16 09:39:23.770] [DEBUG] - Found for obagent-param-1.3.0 for obagent-4.2.2 [2024-10-16 09:39:23.770] [DEBUG] - Applying obagent-param-1.3.0 for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:23.858] [DEBUG] - Search param plugin for ocp-express [2024-10-16 09:39:23.859] [DEBUG] - Found for ocp-express-param-4.1.0 for ocp-express-4.2.2 [2024-10-16 09:39:23.859] [DEBUG] - Applying ocp-express-param-4.1.0 for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:24.121] [INFO] Open ssh connection [2024-10-16 09:39:24.127] [DEBUG] - host: 22.22.22.5, port: 10022, user: root, password: None [2024-10-16 09:39:24.796] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:24.796] [DEBUG] - import start_check [2024-10-16 09:39:24.804] [DEBUG] - add start_check ref count to 1 [2024-10-16 09:39:24.805] [INFO] Check before start observer [2024-10-16 09:39:24.807] [DEBUG] -- root@22.22.22.5 execute: ls /opt/oceanbase/store/clog/tenant_1/ [2024-10-16 09:39:24.822] [DEBUG] -- exited code 0 [2024-10-16 09:39:24.823] [DEBUG] -- root@22.22.22.5 execute: cat /opt/oceanbase/run/observer.pid [2024-10-16 09:39:24.897] [DEBUG] -- exited code 1, error output: [2024-10-16 09:39:24.898] [DEBUG] cat: /opt/oceanbase/run/observer.pid: No such file or directory [2024-10-16 09:39:24.898] [DEBUG] [2024-10-16 09:39:24.898] [DEBUG] -- 22.22.22.5 port check [2024-10-16 09:39:24.898] [DEBUG] -- root@22.22.22.5 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B41' | awk -F' ' '{print $3}' | uniq [2024-10-16 09:39:24.974] [DEBUG] -- exited code 0 [2024-10-16 09:39:24.975] [DEBUG] -- root@22.22.22.5 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B42' | awk -F' ' '{print $3}' | uniq [2024-10-16 09:39:25.044] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.044] [DEBUG] -- root@22.22.22.5 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B46' | awk -F' ' '{print $3}' | uniq [2024-10-16 09:39:25.113] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.113] [DEBUG] -- root@22.22.22.5 execute: ls /opt/oceanbase/store/sstable/block_file [2024-10-16 09:39:25.180] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.181] [DEBUG] -- root@22.22.22.5 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-10-16 09:39:25.251] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.252] [DEBUG] -- root@22.22.22.5 execute: ulimit -a [2024-10-16 09:39:25.310] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.311] [WARNING] OBD-1007: (22.22.22.5) The recommended number of stack size is unlimited (Current value: 8192) [2024-10-16 09:39:25.311] [DEBUG] -- root@22.22.22.5 execute: sysctl -a [2024-10-16 09:39:25.500] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.534] [WARNING] OBD-1017: (22.22.22.5) The value of the "vm.min_free_kbytes" must be within [32768, 2097152] (Current value: 17301504, Recommended value: 2097152) [2024-10-16 09:39:25.534] [WARNING] OBD-1017: (22.22.22.5) The value of the "vm.overcommit_memory" must be 0 (Current value: 1, Recommended value: 0) [2024-10-16 09:39:25.535] [WARNING] OBD-1017: (22.22.22.5) The value of the "fs.file-max" must be greater than 6573688 (Current value: 2097152, Recommended value: 6573688) [2024-10-16 09:39:25.535] [DEBUG] -- root@22.22.22.5 execute: cat /proc/meminfo [2024-10-16 09:39:25.562] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.563] [DEBUG] -- root@22.22.22.5 execute: df --block-size=1024 [2024-10-16 09:39:25.622] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.623] [DEBUG] -- get disk info for path /, total: 2371009208320 avail: 1691068047360 [2024-10-16 09:39:25.623] [DEBUG] -- get disk info for path /dev, total: 67108864 avail: 67108864 [2024-10-16 09:39:25.623] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:25.624] [DEBUG] -- get disk info for path /opt, total: 2371009208320 avail: 1691068047360 [2024-10-16 09:39:25.624] [DEBUG] -- get disk info for path /dev/shm, total: 67108864 avail: 67108864 [2024-10-16 09:39:25.624] [DEBUG] -- get disk info for path /run/secrets/kubernetes.io/serviceaccount, total: 34359738368 avail: 34359541760 [2024-10-16 09:39:25.624] [DEBUG] -- get disk info for path /proc/asound, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:25.624] [DEBUG] -- get disk info for path /proc/acpi, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:25.624] [DEBUG] -- get disk info for path /proc/scsi, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:25.624] [DEBUG] -- get disk info for path /sys/firmware, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:25.624] [DEBUG] -- disk: {'/': {'total': 2371009208320, 'avail': 1691068047360, 'need': 0}, '/dev': {'total': 67108864, 'avail': 67108864, 'need': 0}, '/sys/fs/cgroup': {'total': 273667063808, 'avail': 273667063808, 'need': 0}, '/opt': {'total': 2371009208320, 'avail': 1691068047360, 'need': 0}, '/dev/shm': {'total': 67108864, 'avail': 67108864, 'need': 0}, '/run/secrets/kubernetes.io/serviceaccount': {'total': 34359738368, 'avail': 34359541760, 'need': 0}, '/proc/asound': {'total': 273667063808, 'avail': 273667063808, 'need': 0}, '/proc/acpi': {'total': 273667063808, 'avail': 273667063808, 'need': 0}, '/proc/scsi': {'total': 273667063808, 'avail': 273667063808, 'need': 0}, '/sys/firmware': {'total': 273667063808, 'avail': 273667063808, 'need': 0}} [2024-10-16 09:39:25.624] [DEBUG] -- root@22.22.22.5 execute: date +%s%N [2024-10-16 09:39:25.682] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.682] [DEBUG] -- 22.22.22.5 time delta -1.605224609375 [2024-10-16 09:39:25.720] [INFO] [WARN] OBD-1007: (22.22.22.5) The recommended number of stack size is unlimited (Current value: 8192) [2024-10-16 09:39:25.720] [INFO] [WARN] OBD-1017: (22.22.22.5) The value of the "vm.min_free_kbytes" must be within [32768, 2097152] (Current value: 17301504, Recommended value: 2097152) [2024-10-16 09:39:25.720] [INFO] [WARN] OBD-1017: (22.22.22.5) The value of the "vm.overcommit_memory" must be 0 (Current value: 1, Recommended value: 0) [2024-10-16 09:39:25.720] [INFO] [WARN] OBD-1017: (22.22.22.5) The value of the "fs.file-max" must be greater than 6573688 (Current value: 2097152, Recommended value: 6573688) [2024-10-16 09:39:25.720] [INFO] [2024-10-16 09:39:25.722] [DEBUG] - sub start_check ref count to 0 [2024-10-16 09:39:25.722] [DEBUG] - export start_check [2024-10-16 09:39:25.722] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.2.2.0 result: True [2024-10-16 09:39:25.722] [DEBUG] - Call obproxy-ce-py_script_start_check-4.2.3 for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:25.722] [DEBUG] - import start_check [2024-10-16 09:39:25.725] [DEBUG] - add start_check ref count to 1 [2024-10-16 09:39:25.725] [INFO] Check before start obproxy [2024-10-16 09:39:25.726] [WARNING] OBD-4521: The config observer_sys_password in obproxy-ce did not take effect, please config it in oceanbase-ce [2024-10-16 09:39:25.726] [DEBUG] -- root@22.22.22.5 execute: cat /opt/obproxy/run/obproxy-22.22.22.5-2883.pid [2024-10-16 09:39:25.752] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.752] [DEBUG] -- 22.22.22.5 port check [2024-10-16 09:39:25.752] [DEBUG] -- root@22.22.22.5 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B43' | awk -F' ' '{print $3}' | uniq [2024-10-16 09:39:25.824] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.825] [DEBUG] -- root@22.22.22.5 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B44' | awk -F' ' '{print $3}' | uniq [2024-10-16 09:39:25.894] [DEBUG] -- exited code 0 [2024-10-16 09:39:25.988] [INFO] [WARN] OBD-4521: The config observer_sys_password in obproxy-ce did not take effect, please config it in oceanbase-ce [2024-10-16 09:39:25.988] [INFO] [2024-10-16 09:39:25.988] [DEBUG] - sub start_check ref count to 0 [2024-10-16 09:39:25.988] [DEBUG] - export start_check [2024-10-16 09:39:25.988] [DEBUG] - plugin obproxy-ce-py_script_start_check-4.2.3 result: True [2024-10-16 09:39:25.988] [DEBUG] - Call obagent-py_script_start_check-1.3.0 for obagent-4.2.2-100000042024011120.el8-d8172a2bf3811219777124490558bf800dffef87 [2024-10-16 09:39:25.989] [DEBUG] - import start_check [2024-10-16 09:39:25.992] [DEBUG] - add start_check ref count to 1 [2024-10-16 09:39:25.992] [INFO] Check before start obagent [2024-10-16 09:39:25.996] [DEBUG] -- root@22.22.22.5 execute: cat /opt/obagent/run/ob_agentd.pid [2024-10-16 09:39:26.009] [DEBUG] -- exited code 1, error output: [2024-10-16 09:39:26.009] [DEBUG] cat: /opt/obagent/run/ob_agentd.pid: No such file or directory [2024-10-16 09:39:26.009] [DEBUG] [2024-10-16 09:39:26.010] [DEBUG] -- 22.22.22.5 port check [2024-10-16 09:39:26.010] [DEBUG] -- root@22.22.22.5 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1F99' | awk -F' ' '{print $3}' | uniq [2024-10-16 09:39:26.083] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.084] [DEBUG] -- root@22.22.22.5 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1F98' | awk -F' ' '{print $3}' | uniq [2024-10-16 09:39:26.155] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.257] [DEBUG] - sub start_check ref count to 0 [2024-10-16 09:39:26.258] [DEBUG] - export start_check [2024-10-16 09:39:26.258] [DEBUG] - plugin obagent-py_script_start_check-1.3.0 result: True [2024-10-16 09:39:26.258] [DEBUG] - Call ocp-express-py_script_start_check-4.2.2 for ocp-express-4.2.2-100000022024011120.el8-44a58e92455e3bb1ff8b3cd85c4e1e1cced4ea3e [2024-10-16 09:39:26.258] [DEBUG] - import start_check [2024-10-16 09:39:26.263] [DEBUG] - add start_check ref count to 1 [2024-10-16 09:39:26.264] [INFO] Check before start ocp-express [2024-10-16 09:39:26.267] [DEBUG] -- root@22.22.22.5 execute: cat /opt/ocpexpress/run/ocp-express.pid [2024-10-16 09:39:26.280] [DEBUG] -- exited code 1, error output: [2024-10-16 09:39:26.280] [DEBUG] cat: /opt/ocpexpress/run/ocp-express.pid: No such file or directory [2024-10-16 09:39:26.280] [DEBUG] [2024-10-16 09:39:26.281] [DEBUG] -- root@22.22.22.5 execute: bash -c 'cat /proc/net/{udp*,tcp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FF4' | awk -F' ' '{print $3}' | uniq [2024-10-16 09:39:26.354] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.355] [DEBUG] -- root@22.22.22.5 append '/opt/ocpexpress/jre/bin:' to PATH [2024-10-16 09:39:26.355] [DEBUG] -- root@22.22.22.5 execute: java -version [2024-10-16 09:39:26.474] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.476] [DEBUG] -- root@22.22.22.5 execute: cat /proc/meminfo [2024-10-16 09:39:26.541] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.542] [DEBUG] -- root@22.22.22.5 execute: df --block-size=1024 [2024-10-16 09:39:26.602] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.603] [DEBUG] -- get disk info for path /, total: 2371009208320 avail: 1691068030976 [2024-10-16 09:39:26.603] [DEBUG] -- get disk info for path /dev, total: 67108864 avail: 67108864 [2024-10-16 09:39:26.603] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:26.604] [DEBUG] -- get disk info for path /opt, total: 2371009208320 avail: 1691068030976 [2024-10-16 09:39:26.604] [DEBUG] -- get disk info for path /dev/shm, total: 67108864 avail: 67108864 [2024-10-16 09:39:26.604] [DEBUG] -- get disk info for path /run/secrets/kubernetes.io/serviceaccount, total: 34359738368 avail: 34359541760 [2024-10-16 09:39:26.604] [DEBUG] -- get disk info for path /proc/asound, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:26.604] [DEBUG] -- get disk info for path /proc/acpi, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:26.604] [DEBUG] -- get disk info for path /proc/scsi, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:26.604] [DEBUG] -- get disk info for path /sys/firmware, total: 273667063808 avail: 273667063808 [2024-10-16 09:39:26.604] [DEBUG] -- root@22.22.22.5 execute: df --block-size=1024 /opt/ocpexpress/log [2024-10-16 09:39:26.662] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.662] [DEBUG] -- get disk info for path /opt, total: 2371009208320 avail: 1691068022784 [2024-10-16 09:39:26.790] [DEBUG] - sub start_check ref count to 0 [2024-10-16 09:39:26.790] [DEBUG] - export start_check [2024-10-16 09:39:26.790] [DEBUG] - plugin ocp-express-py_script_start_check-4.2.2 result: True [2024-10-16 09:39:26.790] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:26.790] [DEBUG] - import start [2024-10-16 09:39:26.794] [DEBUG] - add start ref count to 1 [2024-10-16 09:39:26.794] [INFO] Start observer [2024-10-16 09:39:26.795] [DEBUG] -- root@22.22.22.5 execute: ls /opt/oceanbase/store/clog/tenant_1/ [2024-10-16 09:39:26.808] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.809] [DEBUG] -- root@22.22.22.5 execute: cat /opt/oceanbase/run/observer.pid [2024-10-16 09:39:26.882] [DEBUG] -- exited code 1, error output: [2024-10-16 09:39:26.882] [DEBUG] cat: /opt/oceanbase/run/observer.pid: No such file or directory [2024-10-16 09:39:26.882] [DEBUG] [2024-10-16 09:39:26.883] [DEBUG] -- 22.22.22.5 start command construction [2024-10-16 09:39:26.883] [DEBUG] -- starting 22.22.22.5 observer [2024-10-16 09:39:26.883] [DEBUG] -- root@22.22.22.5 set env LD_LIBRARY_PATH to '/opt/oceanbase/lib:' [2024-10-16 09:39:26.884] [DEBUG] -- root@22.22.22.5 execute: cd /opt/oceanbase; /opt/oceanbase/bin/observer -r '22.22.22.5:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'cluster1' -c 1719387089 -d '/opt/oceanbase/store' -l 'WARN' -I '22.22.22.5' -o __min_full_resource_pool_memory=1073741824,scenario='htap',enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit='24240M',memory_size='16144M',datafile_size='8G',datafile_next='2G',log_disk_size='12G',trace_log_slow_query_watermark='300ms',system_memory='2048M',cpu_count=6 [2024-10-16 09:39:26.981] [DEBUG] -- exited code 0 [2024-10-16 09:39:26.982] [DEBUG] -- root@22.22.22.5 delete env LD_LIBRARY_PATH [2024-10-16 09:39:27.056] [DEBUG] -- start_obshell: True [2024-10-16 09:39:27.057] [DEBUG] -- root@22.22.22.5 execute: cat /opt/oceanbase/run/obshell.pid [2024-10-16 09:39:27.070] [DEBUG] -- exited code 1, error output: [2024-10-16 09:39:27.070] [DEBUG] cat: /opt/oceanbase/run/obshell.pid: No such file or directory [2024-10-16 09:39:27.070] [DEBUG] [2024-10-16 09:39:27.070] [DEBUG] -- root@22.22.22.5 set env OB_ROOT_PASSWORD to '''' [2024-10-16 09:39:27.071] [DEBUG] -- start obshell: cd /opt/oceanbase; /opt/oceanbase/bin/obshell admin start --ip 22.22.22.5 --port 2886 [2024-10-16 09:39:27.071] [DEBUG] -- root@22.22.22.5 execute: cd /opt/oceanbase; /opt/oceanbase/bin/obshell admin start --ip 22.22.22.5 --port 2886 [2024-10-16 09:39:29.178] [DEBUG] -- exited code 0 [2024-10-16 09:39:29.179] [INFO] observer program health check [2024-10-16 09:39:32.183] [DEBUG] -- 22.22.22.5 program health check [2024-10-16 09:39:32.183] [DEBUG] -- root@22.22.22.5 execute: cat /opt/oceanbase/run/observer.pid [2024-10-16 09:39:32.195] [DEBUG] -- exited code 0 [2024-10-16 09:39:32.195] [DEBUG] -- root@22.22.22.5 execute: ls /proc/29009 [2024-10-16 09:39:32.261] [DEBUG] -- exited code 0 [2024-10-16 09:39:32.261] [DEBUG] -- 22.22.22.5 observer[pid: 29009] started [2024-10-16 09:39:32.310] [INFO] obshell program health check [2024-10-16 09:39:32.311] [DEBUG] -- root@22.22.22.5 execute: cat /opt/oceanbase/run/obshell.pid [2024-10-16 09:39:32.323] [DEBUG] -- exited code 0 [2024-10-16 09:39:32.324] [DEBUG] -- root@22.22.22.5 execute: ls /proc/29101 [2024-10-16 09:39:32.390] [DEBUG] -- exited code 0 [2024-10-16 09:39:32.391] [DEBUG] -- 22.22.22.5 obshell[pid: 29101] started [2024-10-16 09:39:32.442] [DEBUG] -- need_bootstrap: False [2024-10-16 09:39:32.443] [DEBUG] - sub start ref count to 0 [2024-10-16 09:39:32.443] [DEBUG] - export start [2024-10-16 09:39:32.443] [DEBUG] - plugin oceanbase-ce-py_script_start-4.2.2.0 result: True [2024-10-16 09:39:32.443] [DEBUG] - Call oceanbase-ce-py_script_connect-4.2.2.0 for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:32.443] [DEBUG] - import connect [2024-10-16 09:39:32.545] [DEBUG] - add connect ref count to 1 [2024-10-16 09:39:32.545] [DEBUG] -- connect obshell (22.22.22.5:2886) [2024-10-16 09:39:32.546] [INFO] Connect to observer [2024-10-16 09:39:32.547] [DEBUG] -- connect 22.22.22.5 -P2881 -uroot -p [2024-10-16 09:39:35.579] [DEBUG] -- connect 22.22.22.5 -P2881 -uroot -p [2024-10-16 09:39:38.582] [DEBUG] -- connect 22.22.22.5 -P2881 -uroot -p [2024-10-16 09:39:41.584] [DEBUG] -- connect 22.22.22.5 -P2881 -uroot -p [2024-10-16 09:39:44.588] [DEBUG] -- connect 22.22.22.5 -P2881 -uroot -p [2024-10-16 09:39:47.593] [DEBUG] -- connect 22.22.22.5 -P2881 -uroot -p [2024-10-16 09:39:50.597] [DEBUG] -- connect 22.22.22.5 -P2881 -uroot -p [2024-10-16 09:39:50.599] [DEBUG] -- execute sql: select 1. args: None [2024-10-16 09:39:50.625] [DEBUG] - sub connect ref count to 0 [2024-10-16 09:39:50.625] [DEBUG] - export connect [2024-10-16 09:39:50.625] [DEBUG] - plugin oceanbase-ce-py_script_connect-4.2.2.0 result: True [2024-10-16 09:39:50.625] [DEBUG] - Call obproxy-ce-py_script_start-4.2.3 for obproxy-ce-4.3.1.0-4.el8-446ebd84845911de07dc90db93e55c62338adb9b [2024-10-16 09:39:50.625] [DEBUG] - import start [2024-10-16 09:39:50.628] [DEBUG] - add start ref count to 1 [2024-10-16 09:39:50.631] [INFO] Start obproxy [2024-10-16 09:39:50.631] [DEBUG] -- root@22.22.22.5 execute: ls /opt/obproxy/etc/obproxy_config.bin [2024-10-16 09:39:50.645] [DEBUG] -- exited code 2, error output: [2024-10-16 09:39:50.645] [DEBUG] ls: cannot access '/opt/obproxy/etc/obproxy_config.bin': No such file or directory [2024-10-16 09:39:50.645] [DEBUG] [2024-10-16 09:39:50.645] [DEBUG] -- 22.22.22.5 port check [2024-10-16 09:39:50.646] [DEBUG] -- root@22.22.22.5 execute: cat /opt/obproxy/run/obproxy-22.22.22.5-2883.pid [2024-10-16 09:39:50.702] [DEBUG] -- exited code 0 [2024-10-16 09:39:50.703] [DEBUG] -- starting 22.22.22.5 obproxy [2024-10-16 09:39:50.703] [DEBUG] -- root@22.22.22.5 set env LD_LIBRARY_PATH to '/opt/obproxy/lib:' [2024-10-16 09:39:50.703] [DEBUG] -- root@22.22.22.5 execute: cd /opt/obproxy; /opt/obproxy/bin/obproxy -o client_session_id_version=2,proxy_id=3585,obproxy_sys_password='',enable_strict_kernel_release=False,ip_listen_mode=3,enable_cluster_checkout=False,skip_proxy_sys_private_check=True,proxy_id=3585,client_session_id_version=2 --listen_port 2883 --prometheus_listen_port 2884 --rs_list '22.22.22.5:2881' --cluster_name 'cluster1' [2024-10-16 09:39:50.772] [DEBUG] -- exited code 0 [2024-10-16 09:39:50.772] [DEBUG] -- root@22.22.22.5 delete env LD_LIBRARY_PATH [2024-10-16 09:39:50.773] [DEBUG] -- root@22.22.22.5 execute: ps -aux | grep -e '/opt/obproxy/bin/obproxy -o client_session_id_version=2,proxy_id=3585,obproxy_sys_password=bad1133d69d244bdede506ea90ef8fe8630b0f68,enable_strict_kernel_release=False,ip_listen_mode=3,enable_cluster_checkout=False,skip_proxy_sys_private_check=True,proxy_id=3585,client_session_id_version=2 --listen_port 2883 --prometheus_listen_port 2884 --rs_list 22.22.22.5:2881 --cluster_name cluster1$' | grep -v grep | awk '{print $2}' > /opt/obproxy/run/obproxy-22.22.22.5-2883.pid [2024-10-16 09:39:50.838] [DEBUG] -- exited code 0 [2024-10-16 09:39:50.893] [INFO] obproxy program health check [2024-10-16 09:39:50.893] [DEBUG] -- 22.22.22.5 program health check [2024-10-16 09:39:50.894] [DEBUG] -- root@22.22.22.5 execute: cat /opt/obproxy/run/obproxy-22.22.22.5-2883.pid [2024-10-16 09:39:50.912] [DEBUG] -- exited code 0 [2024-10-16 09:39:51.024] [WARNING] failed to start 22.22.22.5 obproxy [2024-10-16 09:39:51.025] [DEBUG] - sub start ref count to 0 [2024-10-16 09:39:51.025] [DEBUG] - export start [2024-10-16 09:39:51.025] [DEBUG] - plugin obproxy-ce-py_script_start-4.2.3 result: False [2024-10-16 09:39:51.025] [ERROR] obproxy-ce start failed [2024-10-16 09:39:51.025] [DEBUG] - Call oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.4.0-100000082024070810.el8-8b1365b55251aae29758571ac6009989f2262ca9 [2024-10-16 09:39:51.025] [DEBUG] - import display [2024-10-16 09:39:51.027] [DEBUG] - add display ref count to 1 [2024-10-16 09:39:51.027] [INFO] Wait for observer init [2024-10-16 09:39:51.028] [DEBUG] -- execute sql: select * from oceanbase.__all_server. args: None [2024-10-16 09:39:51.031] [INFO] +----------------------------------------------+ [2024-10-16 09:39:51.031] [INFO] | observer | [2024-10-16 09:39:51.032] [INFO] +------------+---------+------+-------+--------+ [2024-10-16 09:39:51.032] [INFO] | ip | version | port | zone | status | [2024-10-16 09:39:51.032] [INFO] +------------+---------+------+-------+--------+ [2024-10-16 09:39:51.032] [INFO] | 22.22.22.5 | 4.2.4.0 | 2881 | zone1 | ACTIVE | [2024-10-16 09:39:51.032] [INFO] +------------+---------+------+-------+--------+ [2024-10-16 09:39:51.032] [INFO] obclient -h22.22.22.5 -P2881 -uroot -p'' -Doceanbase -A [2024-10-16 09:39:51.159] [INFO] +----------------------------------------------+ [2024-10-16 09:39:51.159] [INFO] | observer | [2024-10-16 09:39:51.159] [INFO] +------------+---------+------+-------+--------+ [2024-10-16 09:39:51.159] [INFO] | ip | version | port | zone | status | [2024-10-16 09:39:51.159] [INFO] +------------+---------+------+-------+--------+ [2024-10-16 09:39:51.159] [INFO] | 22.22.22.5 | 4.2.4.0 | 2881 | zone1 | ACTIVE | [2024-10-16 09:39:51.159] [INFO] +------------+---------+------+-------+--------+ [2024-10-16 09:39:51.159] [INFO] obclient -h22.22.22.5 -P2881 -uroot -p'' -Doceanbase -A [2024-10-16 09:39:51.159] [INFO] [2024-10-16 09:39:51.159] [DEBUG] -- execute sql: select unix_timestamp(gmt_create) as gmt_create from oceanbase.__all_virtual_sys_variable limit 1. args: None [2024-10-16 09:39:51.170] [INFO] cluster unique id: d4ca6539-f6d2-56ed-8af1-382a01711489-191504bb621-00040204 [2024-10-16 09:39:51.170] [INFO] [2024-10-16 09:39:51.171] [DEBUG] - sub display ref count to 0 [2024-10-16 09:39:51.171] [DEBUG] - export display [2024-10-16 09:39:51.171] [DEBUG] - plugin oceanbase-ce-py_script_display-3.1.0 result: True [2024-10-16 09:39:51.173] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2024-10-16 09:39:51.173] [INFO] Trace ID: 86402d2e-8ba2-11ef-b96c-b04fa692c9e3 [2024-10-16 09:39:51.173] [INFO] If you want to view detailed obd logs, please run: obd display-trace 86402d2e-8ba2-11ef-b96c-b04fa692c9e3 [2024-10-16 09:39:51.173] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 4 [2024-10-16 09:39:51.173] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 3 [2024-10-16 09:39:51.173] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 2 [2024-10-16 09:39:51.174] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1 [2024-10-16 09:39:51.174] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0 [2024-10-16 09:39:51.174] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo [2024-10-16 09:39:51.174] [DEBUG] - exclusive lock /root/.obd/lock/deploy_cluster1 release, count 0 [2024-10-16 09:39:51.174] [DEBUG] - unlock /root/.obd/lock/deploy_cluster1 [2024-10-16 09:39:51.174] [DEBUG] - share lock /root/.obd/lock/global release, count 0 [2024-10-16 09:39:51.174] [DEBUG] - unlock /root/.obd/lock/global