[2024-05-29 00:11:17.754] [DEBUG] - Get deploy config [2024-05-29 00:11:17.755] [INFO] Search plugins [2024-05-29 00:11:17.755] [DEBUG] - Searching start_check plugin for components ... [2024-05-29 00:11:17.755] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.755] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-05-29 00:11:17.755] [DEBUG] - Searching start_check plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-05-29 00:11:17.755] [DEBUG] - Found for obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.2.1.0 [2024-05-29 00:11:17.755] [DEBUG] - Searching start_check plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-05-29 00:11:17.755] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-4.2.2 [2024-05-29 00:11:17.756] [DEBUG] - Searching create_tenant plugin for components ... [2024-05-29 00:11:17.756] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.756] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.2.0 [2024-05-29 00:11:17.756] [DEBUG] - Searching create_tenant plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-05-29 00:11:17.756] [DEBUG] - No such create_tenant plugin for obproxy-ce-4.2.1.0 [2024-05-29 00:11:17.756] [DEBUG] - Searching create_tenant plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-05-29 00:11:17.756] [DEBUG] - No such create_tenant plugin for obagent-4.2.2 [2024-05-29 00:11:17.756] [DEBUG] - Searching start plugin for components ... [2024-05-29 00:11:17.757] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.757] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-05-29 00:11:17.757] [DEBUG] - Searching start plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-05-29 00:11:17.757] [DEBUG] - Found for obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.2.1.0 [2024-05-29 00:11:17.757] [DEBUG] - Searching start plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-05-29 00:11:17.758] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-4.2.2 [2024-05-29 00:11:17.758] [DEBUG] - Searching connect plugin for components ... [2024-05-29 00:11:17.758] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.758] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-05-29 00:11:17.758] [DEBUG] - Searching connect plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-05-29 00:11:17.758] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.2.1.0 [2024-05-29 00:11:17.758] [DEBUG] - Searching connect plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-05-29 00:11:17.759] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-4.2.2 [2024-05-29 00:11:17.759] [DEBUG] - Searching bootstrap plugin for components ... [2024-05-29 00:11:17.759] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.759] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-05-29 00:11:17.759] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-05-29 00:11:17.759] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.2.1.0 [2024-05-29 00:11:17.759] [DEBUG] - Searching bootstrap plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-05-29 00:11:17.760] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-4.2.2 [2024-05-29 00:11:17.760] [DEBUG] - Searching display plugin for components ... [2024-05-29 00:11:17.760] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.760] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.2.0 [2024-05-29 00:11:17.760] [DEBUG] - Searching display plugin for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-05-29 00:11:17.760] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.2.1.0 [2024-05-29 00:11:17.760] [DEBUG] - Searching display plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-05-29 00:11:17.760] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-4.2.2 [2024-05-29 00:11:17.761] [INFO] Load cluster param plugin [2024-05-29 00:11:17.761] [DEBUG] - Get local repository oceanbase-ce-4.2.2.0-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.761] [DEBUG] - Get local repository obproxy-ce-4.2.1.0-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-05-29 00:11:17.761] [DEBUG] - Get local repository obagent-4.2.2-19739a07a12eab736aff86ecf357b1ae660b554e [2024-05-29 00:11:17.761] [DEBUG] - Searching param plugin for components ... [2024-05-29 00:11:17.761] [DEBUG] - Search param plugin for oceanbase-ce [2024-05-29 00:11:17.761] [DEBUG] - Found for oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.2.0 [2024-05-29 00:11:17.761] [DEBUG] - Applying oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.762] [DEBUG] - Search param plugin for obproxy-ce [2024-05-29 00:11:17.762] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0 [2024-05-29 00:11:17.762] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.2.1.0-11.el7-0aed4b782120e4248b749f67be3d2cc82cdcb70d [2024-05-29 00:11:17.762] [DEBUG] - Search param plugin for obagent [2024-05-29 00:11:17.762] [DEBUG] - Found for obagent-param-1.3.0 for obagent-4.2.2 [2024-05-29 00:11:17.762] [DEBUG] - Applying obagent-param-1.3.0 for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-05-29 00:11:17.762] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:17.763] [INFO] Check before start observer [2024-05-29 00:11:17.763] [DEBUG] -- admin@10.32.2.71 execute: ls /data/oceanbase_data/clog/tenant_1/ [2024-05-29 00:11:17.780] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:17.780] [DEBUG] ls: cannot access /data/oceanbase_data/clog/tenant_1/: No such file or directory [2024-05-29 00:11:17.780] [DEBUG] [2024-05-29 00:11:17.781] [DEBUG] -- admin@10.32.2.71 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:17.836] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:17.836] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:17.836] [DEBUG] [2024-05-29 00:11:17.836] [DEBUG] -- 10.32.2.71 port check [2024-05-29 00:11:17.836] [DEBUG] -- admin@10.32.2.71 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:17.900] [DEBUG] -- exited code 0 [2024-05-29 00:11:17.900] [DEBUG] -- admin@10.32.2.71 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:17.963] [DEBUG] -- exited code 0 [2024-05-29 00:11:17.963] [DEBUG] -- admin@10.32.2.71 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '4702200A:0B46' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.027] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.027] [DEBUG] -- admin@10.32.2.71 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B46' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.091] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.091] [DEBUG] -- admin@10.32.2.71 execute: ls /data/oceanbase_data/sstable/block_file [2024-05-29 00:11:18.146] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:18.146] [DEBUG] ls: cannot access /data/oceanbase_data/sstable/block_file: No such file or directory [2024-05-29 00:11:18.146] [DEBUG] [2024-05-29 00:11:18.147] [DEBUG] -- admin@10.32.2.72 execute: ls /data/oceanbase_data/clog/tenant_1/ [2024-05-29 00:11:18.164] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:18.164] [DEBUG] ls: cannot access /data/oceanbase_data/clog/tenant_1/: No such file or directory [2024-05-29 00:11:18.164] [DEBUG] [2024-05-29 00:11:18.164] [DEBUG] -- admin@10.32.2.72 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:18.219] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:18.220] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:18.220] [DEBUG] [2024-05-29 00:11:18.220] [DEBUG] -- 10.32.2.72 port check [2024-05-29 00:11:18.220] [DEBUG] -- admin@10.32.2.72 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.284] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.284] [DEBUG] -- admin@10.32.2.72 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.347] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.347] [DEBUG] -- admin@10.32.2.72 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '4802200A:0B46' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.411] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.411] [DEBUG] -- admin@10.32.2.72 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B46' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.474] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.474] [DEBUG] -- admin@10.32.2.72 execute: ls /data/oceanbase_data/sstable/block_file [2024-05-29 00:11:18.529] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:18.529] [DEBUG] ls: cannot access /data/oceanbase_data/sstable/block_file: No such file or directory [2024-05-29 00:11:18.529] [DEBUG] [2024-05-29 00:11:18.530] [DEBUG] -- admin@10.32.2.73 execute: ls /data/oceanbase_data/clog/tenant_1/ [2024-05-29 00:11:18.546] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:18.546] [DEBUG] ls: cannot access /data/oceanbase_data/clog/tenant_1/: No such file or directory [2024-05-29 00:11:18.546] [DEBUG] [2024-05-29 00:11:18.546] [DEBUG] -- admin@10.32.2.73 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:18.601] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:18.601] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:18.601] [DEBUG] [2024-05-29 00:11:18.601] [DEBUG] -- 10.32.2.73 port check [2024-05-29 00:11:18.601] [DEBUG] -- admin@10.32.2.73 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.666] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.666] [DEBUG] -- admin@10.32.2.73 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.728] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.729] [DEBUG] -- admin@10.32.2.73 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '4902200A:0B46' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.791] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.791] [DEBUG] -- admin@10.32.2.73 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B46' | awk -F' ' '{print $2}' | uniq [2024-05-29 00:11:18.855] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.855] [DEBUG] -- admin@10.32.2.73 execute: ls /data/oceanbase_data/sstable/block_file [2024-05-29 00:11:18.911] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:18.911] [DEBUG] ls: cannot access /data/oceanbase_data/sstable/block_file: No such file or directory [2024-05-29 00:11:18.911] [DEBUG] [2024-05-29 00:11:18.911] [DEBUG] -- admin@10.32.2.73 execute: [ -w /tmp/ ] || [ -w /tmp/obshell ] [2024-05-29 00:11:18.966] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.967] [DEBUG] -- admin@10.32.2.71 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-05-29 00:11:18.982] [DEBUG] -- exited code 0 [2024-05-29 00:11:18.983] [DEBUG] -- admin@10.32.2.71 execute: ulimit -a [2024-05-29 00:11:19.037] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.037] [DEBUG] -- admin@10.32.2.71 execute: sysctl -a [2024-05-29 00:11:19.205] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.223] [DEBUG] -- admin@10.32.2.71 execute: cat /proc/meminfo [2024-05-29 00:11:19.261] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.261] [DEBUG] -- admin@10.32.2.71 execute: df --block-size=1024 [2024-05-29 00:11:19.318] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /, total: 114980552704 avail: 84701077504 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /dev, total: 201990934528 avail: 201990934528 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /dev/shm, total: 202001932288 avail: 202001907712 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /run, total: 202001932288 avail: 201974321152 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 202001932288 avail: 202001932288 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 894779392 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /run/user/0, total: 40400388096 avail: 40400388096 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:19.319] [DEBUG] -- get disk info for path /run/user/1000, total: 40400388096 avail: 40400388096 [2024-05-29 00:11:19.319] [DEBUG] -- admin@10.32.2.71 execute: df --block-size=1024 /data/oceanbase_redo/clog [2024-05-29 00:11:19.374] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.374] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:19.374] [DEBUG] -- admin@10.32.2.71 execute: df --block-size=1024 /data/oceanbase_data [2024-05-29 00:11:19.430] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.430] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:19.430] [DEBUG] -- disk: {'/': {'total': 114980552704, 'avail': 84701077504, 'need': 0}, '/dev': {'total': 201990934528, 'avail': 201990934528, 'need': 0}, '/dev/shm': {'total': 202001932288, 'avail': 202001907712, 'need': 0}, '/run': {'total': 202001932288, 'avail': 201974321152, 'need': 0}, '/sys/fs/cgroup': {'total': 202001932288, 'avail': 202001932288, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 894779392, 'need': 0}, '/run/user/0': {'total': 40400388096, 'avail': 40400388096, 'need': 0}, '/data': {'total': 842199076864, 'avail': 842163900416, 'need': 0}, '/run/user/1000': {'total': 40400388096, 'avail': 40400388096, 'need': 0}} [2024-05-29 00:11:19.430] [WARNING] OBD-1012: (10.32.2.71) clog and data use the same disk (/data) [2024-05-29 00:11:19.430] [DEBUG] -- admin@10.32.2.71 execute: [ -w /tmp/ ] || [ -w /tmp/obshell ] [2024-05-29 00:11:19.485] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.485] [DEBUG] -- admin@10.32.2.72 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-05-29 00:11:19.501] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.501] [DEBUG] -- admin@10.32.2.72 execute: ulimit -a [2024-05-29 00:11:19.556] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.556] [DEBUG] -- admin@10.32.2.72 execute: sysctl -a [2024-05-29 00:11:19.725] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.741] [DEBUG] -- admin@10.32.2.72 execute: cat /proc/meminfo [2024-05-29 00:11:19.780] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.780] [DEBUG] -- admin@10.32.2.72 execute: df --block-size=1024 [2024-05-29 00:11:19.839] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /, total: 114980552704 avail: 98869186560 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /dev, total: 201990934528 avail: 201990934528 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /dev/shm, total: 202001932288 avail: 202001911808 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /run, total: 202001932288 avail: 201974349824 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 202001932288 avail: 202001932288 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 894779392 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /run/user/0, total: 40400388096 avail: 40400388096 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:19.839] [DEBUG] -- get disk info for path /run/user/1000, total: 40400388096 avail: 40400388096 [2024-05-29 00:11:19.839] [DEBUG] -- admin@10.32.2.72 execute: df --block-size=1024 /data/oceanbase_redo/clog [2024-05-29 00:11:19.894] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.894] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:19.894] [DEBUG] -- admin@10.32.2.72 execute: df --block-size=1024 /data/oceanbase_data [2024-05-29 00:11:19.950] [DEBUG] -- exited code 0 [2024-05-29 00:11:19.950] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:19.950] [DEBUG] -- disk: {'/': {'total': 114980552704, 'avail': 98869186560, 'need': 0}, '/dev': {'total': 201990934528, 'avail': 201990934528, 'need': 0}, '/dev/shm': {'total': 202001932288, 'avail': 202001911808, 'need': 0}, '/run': {'total': 202001932288, 'avail': 201974349824, 'need': 0}, '/sys/fs/cgroup': {'total': 202001932288, 'avail': 202001932288, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 894779392, 'need': 0}, '/run/user/0': {'total': 40400388096, 'avail': 40400388096, 'need': 0}, '/data': {'total': 842199076864, 'avail': 842163900416, 'need': 0}, '/run/user/1000': {'total': 40400388096, 'avail': 40400388096, 'need': 0}} [2024-05-29 00:11:19.950] [WARNING] OBD-1012: (10.32.2.72) clog and data use the same disk (/data) [2024-05-29 00:11:19.950] [DEBUG] -- admin@10.32.2.72 execute: [ -w /tmp/ ] || [ -w /tmp/obshell ] [2024-05-29 00:11:20.004] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.004] [DEBUG] -- admin@10.32.2.73 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-05-29 00:11:20.021] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.021] [DEBUG] -- admin@10.32.2.73 execute: ulimit -a [2024-05-29 00:11:20.074] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.075] [DEBUG] -- admin@10.32.2.73 execute: sysctl -a [2024-05-29 00:11:20.245] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.262] [DEBUG] -- admin@10.32.2.73 execute: cat /proc/meminfo [2024-05-29 00:11:20.301] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.301] [DEBUG] -- admin@10.32.2.73 execute: df --block-size=1024 [2024-05-29 00:11:20.359] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.359] [DEBUG] -- get disk info for path /, total: 114980552704 avail: 109639208960 [2024-05-29 00:11:20.359] [DEBUG] -- get disk info for path /dev, total: 201990934528 avail: 201990934528 [2024-05-29 00:11:20.359] [DEBUG] -- get disk info for path /dev/shm, total: 202001932288 avail: 202001911808 [2024-05-29 00:11:20.359] [DEBUG] -- get disk info for path /run, total: 202001932288 avail: 201974349824 [2024-05-29 00:11:20.359] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 202001932288 avail: 202001932288 [2024-05-29 00:11:20.359] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 894783488 [2024-05-29 00:11:20.359] [DEBUG] -- get disk info for path /run/user/0, total: 40400388096 avail: 40400388096 [2024-05-29 00:11:20.359] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:20.360] [DEBUG] -- get disk info for path /run/user/1000, total: 40400388096 avail: 40400388096 [2024-05-29 00:11:20.360] [DEBUG] -- admin@10.32.2.73 execute: df --block-size=1024 /data/oceanbase_redo/clog [2024-05-29 00:11:20.414] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.415] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:20.415] [DEBUG] -- admin@10.32.2.73 execute: df --block-size=1024 /data/oceanbase_data [2024-05-29 00:11:20.469] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.470] [DEBUG] -- get disk info for path /data, total: 842199076864 avail: 842163900416 [2024-05-29 00:11:20.470] [DEBUG] -- disk: {'/': {'total': 114980552704, 'avail': 109639208960, 'need': 0}, '/dev': {'total': 201990934528, 'avail': 201990934528, 'need': 0}, '/dev/shm': {'total': 202001932288, 'avail': 202001911808, 'need': 0}, '/run': {'total': 202001932288, 'avail': 201974349824, 'need': 0}, '/sys/fs/cgroup': {'total': 202001932288, 'avail': 202001932288, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 894783488, 'need': 0}, '/run/user/0': {'total': 40400388096, 'avail': 40400388096, 'need': 0}, '/data': {'total': 842199076864, 'avail': 842163900416, 'need': 0}, '/run/user/1000': {'total': 40400388096, 'avail': 40400388096, 'need': 0}} [2024-05-29 00:11:20.470] [WARNING] OBD-1012: (10.32.2.73) clog and data use the same disk (/data) [2024-05-29 00:11:20.470] [DEBUG] -- admin@10.32.2.71 execute: ping -W 1 -c 1 10.32.2.72 [2024-05-29 00:11:20.490] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.490] [DEBUG] -- admin@10.32.2.71 execute: ping -W 1 -c 1 10.32.2.73 [2024-05-29 00:11:20.554] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.554] [DEBUG] -- admin@10.32.2.72 execute: ping -W 1 -c 1 10.32.2.71 [2024-05-29 00:11:20.572] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.572] [DEBUG] -- admin@10.32.2.72 execute: ping -W 1 -c 1 10.32.2.73 [2024-05-29 00:11:20.629] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.629] [DEBUG] -- admin@10.32.2.73 execute: ping -W 1 -c 1 10.32.2.71 [2024-05-29 00:11:20.647] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.647] [DEBUG] -- admin@10.32.2.73 execute: ping -W 1 -c 1 10.32.2.72 [2024-05-29 00:11:20.703] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.703] [DEBUG] -- admin@10.32.2.71 execute: date +%s%N [2024-05-29 00:11:20.719] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.719] [DEBUG] -- 10.32.2.71 time delta -0.994140625 [2024-05-29 00:11:20.719] [DEBUG] -- admin@10.32.2.72 execute: date +%s%N [2024-05-29 00:11:20.735] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.735] [DEBUG] -- 10.32.2.72 time delta 2.632568359375 [2024-05-29 00:11:20.735] [DEBUG] -- admin@10.32.2.73 execute: date +%s%N [2024-05-29 00:11:20.757] [DEBUG] -- exited code 0 [2024-05-29 00:11:20.758] [DEBUG] -- 10.32.2.73 time delta -7.303466796875 [2024-05-29 00:11:20.758] [INFO] [WARN] OBD-1012: (10.32.2.71) clog and data use the same disk (/data) [2024-05-29 00:11:20.758] [INFO] [WARN] OBD-1012: (10.32.2.72) clog and data use the same disk (/data) [2024-05-29 00:11:20.758] [INFO] [WARN] OBD-1012: (10.32.2.73) clog and data use the same disk (/data) [2024-05-29 00:11:20.758] [INFO] [2024-05-29 00:11:20.758] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.2.0-100010012024022719.el7-60fe842818ef64631dd3bb6f90bf4cab0e737195 [2024-05-29 00:11:20.759] [DEBUG] - import start [2024-05-29 00:11:20.761] [DEBUG] - add start ref count to 1 [2024-05-29 00:11:20.761] [INFO] Start observer [2024-05-29 00:11:20.762] [DEBUG] -- admin@10.32.2.71 execute: ls /data/oceanbase_data/clog/tenant_1/ [2024-05-29 00:11:20.778] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:20.778] [DEBUG] ls: cannot access /data/oceanbase_data/clog/tenant_1/: No such file or directory [2024-05-29 00:11:20.778] [DEBUG] [2024-05-29 00:11:20.778] [DEBUG] -- admin@10.32.2.71 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:20.833] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:20.833] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:20.833] [DEBUG] [2024-05-29 00:11:20.833] [DEBUG] -- 10.32.2.71 start command construction [2024-05-29 00:11:20.833] [DEBUG] -- admin@10.32.2.72 execute: ls /data/oceanbase_data/clog/tenant_1/ [2024-05-29 00:11:20.849] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:20.849] [DEBUG] ls: cannot access /data/oceanbase_data/clog/tenant_1/: No such file or directory [2024-05-29 00:11:20.849] [DEBUG] [2024-05-29 00:11:20.849] [DEBUG] -- admin@10.32.2.72 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:20.907] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:20.907] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:20.907] [DEBUG] [2024-05-29 00:11:20.907] [DEBUG] -- 10.32.2.72 start command construction [2024-05-29 00:11:20.907] [DEBUG] -- admin@10.32.2.73 execute: ls /data/oceanbase_data/clog/tenant_1/ [2024-05-29 00:11:20.924] [DEBUG] -- exited code 2, error output: [2024-05-29 00:11:20.924] [DEBUG] ls: cannot access /data/oceanbase_data/clog/tenant_1/: No such file or directory [2024-05-29 00:11:20.924] [DEBUG] [2024-05-29 00:11:20.924] [DEBUG] -- admin@10.32.2.73 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:20.979] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:20.979] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:20.979] [DEBUG] [2024-05-29 00:11:20.979] [DEBUG] -- 10.32.2.73 start command construction [2024-05-29 00:11:20.980] [DEBUG] -- starting 10.32.2.71 observer [2024-05-29 00:11:20.980] [DEBUG] -- admin@10.32.2.71 set env LD_LIBRARY_PATH to '/home/admin/lahg_oceanbase/oceanbase/lib:' [2024-05-29 00:11:20.980] [DEBUG] -- admin@10.32.2.71 execute: cd /home/admin/lahg_oceanbase/oceanbase; /home/admin/lahg_oceanbase/oceanbase/bin/observer -r '10.32.2.71:2882:2881;10.32.2.72:2882:2881;10.32.2.73:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'lahg_oceanbase' -c 1716911991 -d '/data/oceanbase_data' -I '10.32.2.71' -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit='6G',datafile_size='2G',system_memory='2G',log_disk_size='14G',cpu_count=142 [2024-05-29 00:11:21.247] [DEBUG] -- exited code 0 [2024-05-29 00:11:21.247] [DEBUG] -- admin@10.32.2.71 set env LD_LIBRARY_PATH to '' [2024-05-29 00:11:21.247] [DEBUG] -- starting 10.32.2.72 observer [2024-05-29 00:11:21.247] [DEBUG] -- admin@10.32.2.72 set env LD_LIBRARY_PATH to '/home/admin/lahg_oceanbase/oceanbase/lib:' [2024-05-29 00:11:21.247] [DEBUG] -- admin@10.32.2.72 execute: cd /home/admin/lahg_oceanbase/oceanbase; /home/admin/lahg_oceanbase/oceanbase/bin/observer -r '10.32.2.71:2882:2881;10.32.2.72:2882:2881;10.32.2.73:2882:2881' -p 2881 -P 2882 -z 'zone2' -n 'lahg_oceanbase' -c 1716911991 -d '/data/oceanbase_data' -I '10.32.2.72' -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit='6G',datafile_size='2G',system_memory='2G',log_disk_size='14G',cpu_count=142 [2024-05-29 00:11:21.515] [DEBUG] -- exited code 0 [2024-05-29 00:11:21.515] [DEBUG] -- admin@10.32.2.72 set env LD_LIBRARY_PATH to '' [2024-05-29 00:11:21.515] [DEBUG] -- starting 10.32.2.73 observer [2024-05-29 00:11:21.515] [DEBUG] -- admin@10.32.2.73 set env LD_LIBRARY_PATH to '/home/admin/lahg_oceanbase/oceanbase/lib:' [2024-05-29 00:11:21.515] [DEBUG] -- admin@10.32.2.73 execute: cd /home/admin/lahg_oceanbase/oceanbase; /home/admin/lahg_oceanbase/oceanbase/bin/observer -r '10.32.2.71:2882:2881;10.32.2.72:2882:2881;10.32.2.73:2882:2881' -p 2881 -P 2882 -z 'zone3' -n 'lahg_oceanbase' -c 1716911991 -d '/data/oceanbase_data' -I '10.32.2.73' -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit='6G',datafile_size='2G',system_memory='2G',log_disk_size='14G',cpu_count=142 [2024-05-29 00:11:21.782] [DEBUG] -- exited code 0 [2024-05-29 00:11:21.783] [DEBUG] -- admin@10.32.2.73 set env LD_LIBRARY_PATH to '' [2024-05-29 00:11:21.783] [DEBUG] -- start_obshell: False [2024-05-29 00:11:21.783] [INFO] observer program health check [2024-05-29 00:11:24.786] [DEBUG] -- 10.32.2.71 program health check [2024-05-29 00:11:24.786] [DEBUG] -- admin@10.32.2.71 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:24.803] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:24.803] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:24.803] [DEBUG] [2024-05-29 00:11:24.803] [DEBUG] -- 10.32.2.72 program health check [2024-05-29 00:11:24.803] [DEBUG] -- admin@10.32.2.72 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:24.820] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:24.820] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:24.820] [DEBUG] [2024-05-29 00:11:24.820] [DEBUG] -- 10.32.2.73 program health check [2024-05-29 00:11:24.820] [DEBUG] -- admin@10.32.2.73 execute: cat /home/admin/lahg_oceanbase/oceanbase/run/observer.pid [2024-05-29 00:11:24.837] [DEBUG] -- exited code 1, error output: [2024-05-29 00:11:24.837] [DEBUG] cat: /home/admin/lahg_oceanbase/oceanbase/run/observer.pid: No such file or directory [2024-05-29 00:11:24.838] [DEBUG] [2024-05-29 00:11:24.838] [WARNING] OBD-2002: Failed to start 10.32.2.71 observer [2024-05-29 00:11:24.838] [WARNING] OBD-2002: Failed to start 10.32.2.72 observer [2024-05-29 00:11:24.838] [WARNING] OBD-2002: Failed to start 10.32.2.73 observer [2024-05-29 00:11:24.838] [DEBUG] - sub start ref count to 0 [2024-05-29 00:11:24.838] [DEBUG] - export start [2024-05-29 00:11:24.838] [ERROR] oceanbase-ce start failed