OceanBase无法启动

OceanBase 技术问题
, ,
#1

【 使用环境 】测试环境
【 OB or 其他组件 】OB
【 使用版本 】4.2

如下，我的配置无法启动 oceanbase 集群，我的 yaml 配置文件如下，请帮忙解决一下问题，看起来总是有什么 pid 无法获取的样子，不知道是哪里的问题？

user:
    username: root
    password: 123456
oceanbase-ce:
  version: 4.2.0.0
  servers:
    - name: server1
      ip: 49.52.27.33
    - name: server2
      ip: 49.52.27.34
    - name: server3
      ip: 49.52.27.35
  global:
    memory_limit: 64G
    system_memory: 30G
    home_path: /root/observer
    data_dir: /data/1
    redo_dir: /data/log1
    mysql_port: 2893
    rpc_port: 2894
    datafile_size: 80G
    log_disk_size: 80G
  server1:
    devname: enp95s0f0
    zone: zone1
  server2:
    devname: enp95s0f0
    zone: zone2
  server3:
    devname: enp95s0f0
    zone: zone3
obproxy-ce:
  version: 4.2.0.0
  depends:
    - oceanbase-ce
  servers:
    - 49.52.27.33
  global:
    home_path: /root/obproxy
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    enable_cluster_checkout: false    
    listen_port: 2887
    prometheus_listen_port: 2889
obagent:
  depends:
    - oceanbase-ce
  servers:
    - name: server1
      ip: 49.52.27.33
    - name: server2
      ip: 49.52.27.34
    - name: server3
      ip: 49.52.27.35
  global:
    home_path: /root/obagent
    server_port: 8098
    pprof_port: 8099
    monagent_http_port: 8188
    mgragent_http_port: 8189



[2024-09-23 16:54:31.788] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-09-23 16:54:31.789] [DEBUG] - import start_check
[2024-09-23 16:54:31.807] [DEBUG] - add start_check ref count to 1
[2024-09-23 16:54:31.808] [INFO] Check before start observer
[2024-09-23 16:54:31.811] [DEBUG] -- root@49.52.27.33 execute: ls /data/1/clog/tenant_1/
[2024-09-23 16:54:32.281] [DEBUG] -- exited code 255, error output:
[2024-09-23 16:54:32.281] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-23 16:54:32.281] [DEBUG]
[2024-09-23 16:54:32.282] [DEBUG] -- root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-23 16:54:32.344] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:32.344] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-23 16:54:32.344] [DEBUG]
[2024-09-23 16:54:32.344] [DEBUG] -- server1(49.52.27.33) port check
[2024-09-23 16:54:32.344] [DEBUG] -- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B4D' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:32.450] [DEBUG] -- exited code 0
[2024-09-23 16:54:32.451] [DEBUG] -- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B4E' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:32.556] [DEBUG] -- exited code 0
[2024-09-23 16:54:32.557] [DEBUG] -- root@49.52.27.33 execute: ls /data/1/sstable/block_file
[2024-09-23 16:54:32.624] [DEBUG] -- exited code 255, error output:
[2024-09-23 16:54:32.625] [DEBUG] ls: cannot access /data/1/sstable/block_file: No such file or directory
[2024-09-23 16:54:32.625] [DEBUG]
[2024-09-23 16:54:32.625] [DEBUG] -- root@49.52.27.33 execute: grep -e '^ *enp95s0f0:' /proc/net/dev
[2024-09-23 16:54:32.687] [DEBUG] -- exited code 0
[2024-09-23 16:54:32.691] [DEBUG] -- root@49.52.27.34 execute: ls /data/1/clog/tenant_1/
[2024-09-23 16:54:32.704] [DEBUG] -- exited code 2, error output:
[2024-09-23 16:54:32.705] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-23 16:54:32.705] [DEBUG]
[2024-09-23 16:54:32.705] [DEBUG] -- root@49.52.27.34 execute: cat /root/observer/run/observer.pid
[2024-09-23 16:54:32.756] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:32.757] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-23 16:54:32.757] [DEBUG]
[2024-09-23 16:54:32.757] [DEBUG] -- server2(49.52.27.34) port check
[2024-09-23 16:54:32.757] [DEBUG] -- root@49.52.27.34 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B4D' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:32.813] [DEBUG] -- exited code 0
[2024-09-23 16:54:32.814] [DEBUG] -- root@49.52.27.34 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B4E' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:32.870] [DEBUG] -- exited code 0
[2024-09-23 16:54:32.871] [DEBUG] -- root@49.52.27.34 execute: ls /data/1/sstable/block_file
[2024-09-23 16:54:32.923] [DEBUG] -- exited code 2, error output:
[2024-09-23 16:54:32.923] [DEBUG] ls: cannot access /data/1/sstable/block_file: No such file or directory
[2024-09-23 16:54:32.923] [DEBUG]
[2024-09-23 16:54:32.924] [DEBUG] -- root@49.52.27.34 execute: grep -e '^ *enp95s0f0:' /proc/net/dev
[2024-09-23 16:54:32.975] [DEBUG] -- exited code 0
[2024-09-23 16:54:32.977] [DEBUG] -- root@49.52.27.35 execute: ls /data/1/clog/tenant_1/
[2024-09-23 16:54:32.983] [DEBUG] -- exited code 2, error output:
[2024-09-23 16:54:32.983] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-23 16:54:32.984] [DEBUG]
[2024-09-23 16:54:32.984] [DEBUG] -- root@49.52.27.35 execute: cat /root/observer/run/observer.pid
[2024-09-23 16:54:33.029] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:33.030] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-23 16:54:33.030] [DEBUG]
[2024-09-23 16:54:33.030] [DEBUG] -- server3(49.52.27.35) port check
[2024-09-23 16:54:33.030] [DEBUG] -- root@49.52.27.35 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B4D' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:33.082] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.082] [DEBUG] -- root@49.52.27.35 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B4E' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:33.132] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.132] [DEBUG] -- root@49.52.27.35 execute: ls /data/1/sstable/block_file
[2024-09-23 16:54:33.177] [DEBUG] -- exited code 2, error output:
[2024-09-23 16:54:33.181] [DEBUG] ls: cannot access /data/1/sstable/block_file: No such file or directory
[2024-09-23 16:54:33.181] [DEBUG]
[2024-09-23 16:54:33.181] [DEBUG] -- root@49.52.27.35 execute: grep -e '^ *enp95s0f0:' /proc/net/dev
[2024-09-23 16:54:33.223] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.224] [DEBUG] -- root@49.52.27.33 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2024-09-23 16:54:33.245] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.246] [DEBUG] -- root@49.52.27.33 execute: ulimit -a
[2024-09-23 16:54:33.307] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.308] [DEBUG] -- root@49.52.27.33 execute: sysctl -a
[2024-09-23 16:54:33.512] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.536] [DEBUG] -- root@49.52.27.33 execute: cat /proc/meminfo
[2024-09-23 16:54:33.578] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.580] [DEBUG] -- root@49.52.27.33 execute: df --block-size=1024
[2024-09-23 16:54:33.640] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.641] [DEBUG] -- get disk info for path /dev, total: 201271652352 avail: 201271652352
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /dev/shm, total: 201284481024 avail: 201284481024
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /run, total: 201284481024 avail: 196953059328
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 201284481024 avail: 201284481024
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /, total: 589458046976 avail: 55419645952
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /pg_wal, total: 264088522752 avail: 187537993728
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /pg_data, total: 264088522752 avail: 154882715648
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /data/1, total: 264088522752 avail: 250587041792
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /tikv, total: 257941159936 avail: 54891069440
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /tiflash, total: 257941159936 avail: 67271987200
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /data/log1, total: 264088522752 avail: 250587045888
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /var/lib/ceph, total: 107321753600 avail: 106844774400
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 779599872
[2024-09-23 16:54:33.642] [DEBUG] -- get disk info for path /home, total: 420596469760 avail: 105716301824
[2024-09-23 16:54:33.643] [DEBUG] -- get disk info for path /boot/efi, total: 209489920 avail: 197746688
[2024-09-23 16:54:33.643] [DEBUG] -- get disk info for path /var/lib/docker/overlay2/fa45cf8d16d9f669195bf5d5502aabb4278906fcba5e5c021af7b55cd2e87c54/merged, total: 589458046976 avail: 55419645952
[2024-09-23 16:54:33.643] [DEBUG] -- get disk info for path /run/user/26, total: 40256897024 avail: 40256897024
[2024-09-23 16:54:33.643] [DEBUG] -- get disk info for path /run/user/1003, total: 40256897024 avail: 40256897024
[2024-09-23 16:54:33.643] [DEBUG] -- get disk info for path /run/user/1000, total: 40256897024 avail: 40256897024
[2024-09-23 16:54:33.643] [DEBUG] -- get disk info for path /run/user/0, total: 40256897024 avail: 40256897024
[2024-09-23 16:54:33.643] [DEBUG] -- root@49.52.27.33 execute: df --block-size=1024 /data/log1/clog
[2024-09-23 16:54:33.702] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.703] [DEBUG] -- get disk info for path /data/log1, total: 264088522752 avail: 250587045888
[2024-09-23 16:54:33.703] [DEBUG] -- disk: {'/dev': {'total': 201271652352, 'avail': 201271652352, 'need': 0}, '/dev/shm': {'total': 201284481024, 'avail': 201284481024, 'need': 0}, '/run': {'total': 201284481024, 'avail': 196953059328, 'need': 0}, '/sys/fs/cgroup': {'total': 201284481024, 'avail': 201284481024, 'need': 0}, '/': {'total': 589458046976, 'avail': 55419645952, 'need': 0}, '/pg_wal': {'total': 264088522752, 'avail': 187537993728, 'need': 0}, '/pg_data': {'total': 264088522752, 'avail': 154882715648, 'need': 0}, '/data/1': {'total': 264088522752, 'avail': 250587041792, 'need': 0}, '/tikv': {'total': 257941159936, 'avail': 54891069440, 'need': 0}, '/tiflash': {'total': 257941159936, 'avail': 67271987200, 'need': 0}, '/data/log1': {'total': 264088522752, 'avail': 250587045888, 'need': 0}, '/var/lib/ceph': {'total': 107321753600, 'avail': 106844774400, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 779599872, 'need': 0}, '/home': {'total': 420596469760, 'avail': 105716301824, 'need': 0}, '/boot/efi': {'total': 209489920, 'avail': 197746688, 'need': 0}, '/var/lib/docker/overlay2/fa45cf8d16d9f669195bf5d5502aabb4278906fcba5e5c021af7b55cd2e87c54/merged': {'total': 589458046976, 'avail': 55419645952, 'need': 0}, '/run/user/26': {'total': 40256897024, 'avail': 40256897024, 'need': 0}, '/run/user/1003': {'total': 40256897024, 'avail': 40256897024, 'need': 0}, '/run/user/1000': {'total': 40256897024, 'avail': 40256897024, 'need': 0}, '/run/user/0': {'total': 40256897024, 'avail': 40256897024, 'need': 0}}
[2024-09-23 16:54:33.703] [DEBUG] -- root@49.52.27.34 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2024-09-23 16:54:33.715] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.716] [DEBUG] -- root@49.52.27.34 execute: ulimit -a
[2024-09-23 16:54:33.765] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.766] [WARNING] OBD-1007: (49.52.27.34) The recommended number of stack size is unlimited (Current value: 10240)
[2024-09-23 16:54:33.766] [DEBUG] -- root@49.52.27.34 execute: sysctl -a
[2024-09-23 16:54:33.902] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.927] [DEBUG] -- root@49.52.27.34 execute: cat /proc/meminfo
[2024-09-23 16:54:33.954] [DEBUG] -- exited code 0
[2024-09-23 16:54:33.955] [DEBUG] -- root@49.52.27.34 execute: df --block-size=1024
[2024-09-23 16:54:34.007] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.007] [DEBUG] -- get disk info for path /dev, total: 201270693888 avail: 201270693888
[2024-09-23 16:54:34.007] [DEBUG] -- get disk info for path /dev/shm, total: 201284468736 avail: 201284460544
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /run, total: 201284468736 avail: 197909495808
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 201284468736 avail: 201284468736
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /, total: 589458046976 avail: 120793145344
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 824893440
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /boot/efi, total: 5358223360 avail: 5346328576
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /home, total: 391728594944 avail: 50258780160
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /pg_wal, total: 264088522752 avail: 231041314816
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /pg_data, total: 264088522752 avail: 154883538944
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /data/1, total: 264088522752 avail: 250587041792
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /data/log1, total: 264088522752 avail: 250587045888
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /tikv, total: 257941159936 avail: 62323073024
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /tiflash, total: 257941159936 avail: 68156522496
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /run/user/0, total: 40256897024 avail: 40256897024
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /var/lib/docker/overlay2/5454443b5ad0b80528f4433a00057b56ea03ff20dcce995d53c7ddd6258cc6c5/merged, total: 589458046976 avail: 120793145344
[2024-09-23 16:54:34.008] [DEBUG] -- get disk info for path /run/user/1000, total: 40256897024 avail: 40256897024
[2024-09-23 16:54:34.009] [DEBUG] -- root@49.52.27.34 execute: df --block-size=1024 /data/log1/clog
[2024-09-23 16:54:34.058] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.059] [DEBUG] -- get disk info for path /data/log1, total: 264088522752 avail: 250587045888
[2024-09-23 16:54:34.059] [DEBUG] -- disk: {'/dev': {'total': 201270693888, 'avail': 201270693888, 'need': 0}, '/dev/shm': {'total': 201284468736, 'avail': 201284460544, 'need': 0}, '/run': {'total': 201284468736, 'avail': 197909495808, 'need': 0}, '/sys/fs/cgroup': {'total': 201284468736, 'avail': 201284468736, 'need': 0}, '/': {'total': 589458046976, 'avail': 120793145344, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 824893440, 'need': 0}, '/boot/efi': {'total': 5358223360, 'avail': 5346328576, 'need': 0}, '/home': {'total': 391728594944, 'avail': 50258780160, 'need': 0}, '/pg_wal': {'total': 264088522752, 'avail': 231041314816, 'need': 0}, '/pg_data': {'total': 264088522752, 'avail': 154883538944, 'need': 0}, '/data/1': {'total': 264088522752, 'avail': 250587041792, 'need': 0}, '/data/log1': {'total': 264088522752, 'avail': 250587045888, 'need': 0}, '/tikv': {'total': 257941159936, 'avail': 62323073024, 'need': 0}, '/tiflash': {'total': 257941159936, 'avail': 68156522496, 'need': 0}, '/run/user/0': {'total': 40256897024, 'avail': 40256897024, 'need': 0}, '/var/lib/docker/overlay2/5454443b5ad0b80528f4433a00057b56ea03ff20dcce995d53c7ddd6258cc6c5/merged': {'total': 589458046976, 'avail': 120793145344, 'need': 0}, '/run/user/1000': {'total': 40256897024, 'avail': 40256897024, 'need': 0}}
[2024-09-23 16:54:34.059] [DEBUG] -- root@49.52.27.35 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2024-09-23 16:54:34.066] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.066] [DEBUG] -- root@49.52.27.35 execute: ulimit -a
[2024-09-23 16:54:34.111] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.112] [WARNING] (49.52.27.35) failed to get open files
[2024-09-23 16:54:34.112] [WARNING] (49.52.27.35) failed to get max user processes
[2024-09-23 16:54:34.112] [WARNING] OBD-1007: (49.52.27.35) The recommended number of stack size is unlimited (Current value: 10240)
[2024-09-23 16:54:34.112] [DEBUG] -- root@49.52.27.35 execute: sysctl -a
[2024-09-23 16:54:34.239] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.264] [DEBUG] -- root@49.52.27.35 execute: cat /proc/meminfo
[2024-09-23 16:54:34.285] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.286] [DEBUG] -- root@49.52.27.35 execute: df --block-size=1024
[2024-09-23 16:54:34.331] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /dev, total: 202058973184 avail: 202058973184
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /dev/shm, total: 202076631040 avail: 202076631040
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /run, total: 202076631040 avail: 202047959040
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 202076631040 avail: 202076631040
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /, total: 1477442535424 avail: 469474906112
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /pg_wal, total: 264088522752 avail: 222501351424
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /pg_data, total: 264088522752 avail: 154883227648
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /boot, total: 1063256064 avail: 843157504
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /tikv, total: 257941159936 avail: 50747748352
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /tiflash, total: 257941159936 avail: 61911478272
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /data/log1, total: 264088522752 avail: 250587045888
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /data/1, total: 264088522752 avail: 250587041792
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /mnt/pgdata, total: 3937852530688 avail: 1260564373504
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /boot/efi, total: 209489920 avail: 197586944
[2024-09-23 16:54:34.332] [DEBUG] -- get disk info for path /var/lib/docker/overlay2/8fca5fe929eeee9b3c16d09b1d79fa4249c94feb7cd82a0ccf0cca5682d51050/merged, total: 1477442535424 avail: 469474906112
[2024-09-23 16:54:34.333] [DEBUG] -- get disk info for path /var/lib/docker/overlay2/b0b4b5f44e5799a79b3f1b285fd138d3c09be6a2278cbb55deb7dc01ae7eec1e/merged, total: 1477442535424 avail: 469474906112
[2024-09-23 16:54:34.333] [DEBUG] -- get disk info for path /var/lib/docker/overlay2/4f7657e0d1a6c7f38f8a55e4f6a9a18c3e6d97c0a69ed3861e6720d1760d6b80/merged, total: 1477442535424 avail: 469474906112
[2024-09-23 16:54:34.333] [DEBUG] -- get disk info for path /run/user/0, total: 40415330304 avail: 40415330304
[2024-09-23 16:54:34.333] [DEBUG] -- get disk info for path /run/user/1000, total: 40415330304 avail: 40415330304
[2024-09-23 16:54:34.333] [DEBUG] -- get disk info for path /run/user/5004, total: 40415330304 avail: 40415330304
[2024-09-23 16:54:34.333] [DEBUG] -- get disk info for path /var/lib/docker/overlay2/8fc43c7b6e604b19854b3e2c1db757a14c6806cf939fe31024b3acd59ce2be26/merged, total: 1477442535424 avail: 469474906112
[2024-09-23 16:54:34.333] [DEBUG] -- root@49.52.27.35 execute: df --block-size=1024 /data/log1/clog
[2024-09-23 16:54:34.377] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.377] [DEBUG] -- get disk info for path /data/log1, total: 264088522752 avail: 250587045888
[2024-09-23 16:54:34.377] [DEBUG] -- disk: {'/dev': {'total': 202058973184, 'avail': 202058973184, 'need': 0}, '/dev/shm': {'total': 202076631040, 'avail': 202076631040, 'need': 0}, '/run': {'total': 202076631040, 'avail': 202047959040, 'need': 0}, '/sys/fs/cgroup': {'total': 202076631040, 'avail': 202076631040, 'need': 0}, '/': {'total': 1477442535424, 'avail': 469474906112, 'need': 0}, '/pg_wal': {'total': 264088522752, 'avail': 222501351424, 'need': 0}, '/pg_data': {'total': 264088522752, 'avail': 154883227648, 'need': 0}, '/boot': {'total': 1063256064, 'avail': 843157504, 'need': 0}, '/tikv': {'total': 257941159936, 'avail': 50747748352, 'need': 0}, '/tiflash': {'total': 257941159936, 'avail': 61911478272, 'need': 0}, '/data/log1': {'total': 264088522752, 'avail': 250587045888, 'need': 0}, '/data/1': {'total': 264088522752, 'avail': 250587041792, 'need': 0}, '/mnt/pgdata': {'total': 3937852530688, 'avail': 1260564373504, 'need': 0}, '/boot/efi': {'total': 209489920, 'avail': 197586944, 'need': 0}, '/var/lib/docker/overlay2/8fca5fe929eeee9b3c16d09b1d79fa4249c94feb7cd82a0ccf0cca5682d51050/merged': {'total': 1477442535424, 'avail': 469474906112, 'need': 0}, '/var/lib/docker/overlay2/b0b4b5f44e5799a79b3f1b285fd138d3c09be6a2278cbb55deb7dc01ae7eec1e/merged': {'total': 1477442535424, 'avail': 469474906112, 'need': 0}, '/var/lib/docker/overlay2/4f7657e0d1a6c7f38f8a55e4f6a9a18c3e6d97c0a69ed3861e6720d1760d6b80/merged': {'total': 1477442535424, 'avail': 469474906112, 'need': 0}, '/run/user/0': {'total': 40415330304, 'avail': 40415330304, 'need': 0}, '/run/user/1000': {'total': 40415330304, 'avail': 40415330304, 'need': 0}, '/run/user/5004': {'total': 40415330304, 'avail': 40415330304, 'need': 0}, '/var/lib/docker/overlay2/8fc43c7b6e604b19854b3e2c1db757a14c6806cf939fe31024b3acd59ce2be26/merged': {'total': 1477442535424, 'avail': 469474906112, 'need': 0}}
[2024-09-23 16:54:34.378] [DEBUG] -- root@49.52.27.33 execute: ping -W 1 -c 1 -I enp95s0f0 49.52.27.34
[2024-09-23 16:54:34.402] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.403] [DEBUG] -- root@49.52.27.33 execute: ping -W 1 -c 1 -I enp95s0f0 49.52.27.35
[2024-09-23 16:54:34.467] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.468] [DEBUG] -- root@49.52.27.34 execute: ping -W 1 -c 1 -I enp95s0f0 49.52.27.33
[2024-09-23 16:54:34.482] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.483] [DEBUG] -- root@49.52.27.34 execute: ping -W 1 -c 1 -I enp95s0f0 49.52.27.35
[2024-09-23 16:54:34.534] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.535] [DEBUG] -- root@49.52.27.35 execute: ping -W 1 -c 1 -I enp95s0f0 49.52.27.33
[2024-09-23 16:54:34.543] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.544] [DEBUG] -- root@49.52.27.35 execute: ping -W 1 -c 1 -I enp95s0f0 49.52.27.34
[2024-09-23 16:54:34.589] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.590] [DEBUG] -- root@49.52.27.33 execute: date +%s%N
[2024-09-23 16:54:34.611] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.612] [DEBUG] -- 49.52.27.33 time delta 131305.19848632812
[2024-09-23 16:54:34.612] [DEBUG] -- root@49.52.27.34 execute: date +%s%N
[2024-09-23 16:54:34.624] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.625] [DEBUG] -- 49.52.27.34 time delta 131338.47998046875
[2024-09-23 16:54:34.625] [DEBUG] -- root@49.52.27.35 execute: date +%s%N
[2024-09-23 16:54:34.634] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.634] [DEBUG] -- 49.52.27.35 time delta 131337.15576171875
[2024-09-23 16:54:34.684] [INFO] [WARN] OBD-1007: (49.52.27.34) The recommended number of stack size is unlimited (Current value: 10240)
[2024-09-23 16:54:34.685] [INFO] [WARN] (49.52.27.35) failed to get open files
[2024-09-23 16:54:34.685] [INFO] [WARN] (49.52.27.35) failed to get max user processes
[2024-09-23 16:54:34.685] [INFO] [WARN] OBD-1007: (49.52.27.35) The recommended number of stack size is unlimited (Current value: 10240)
[2024-09-23 16:54:34.685] [INFO]
[2024-09-23 16:54:34.687] [DEBUG] - sub start_check ref count to 0
[2024-09-23 16:54:34.687] [DEBUG] - export start_check
[2024-09-23 16:54:34.688] [DEBUG] - Call obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.2.0.0-7.el7-b3ead2f667d69fe3195fd22002de83df6eb19382
[2024-09-23 16:54:34.688] [DEBUG] - import start_check
[2024-09-23 16:54:34.692] [DEBUG] - add start_check ref count to 1
[2024-09-23 16:54:34.693] [INFO] Check before start obproxy
[2024-09-23 16:54:34.695] [DEBUG] -- root@49.52.27.33 execute: cat /root/obproxy/run/obproxy-49.52.27.33-2887.pid
[2024-09-23 16:54:34.717] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:34.717] [DEBUG] cat: /root/obproxy/run/obproxy-49.52.27.33-2887.pid: No such file or directory
[2024-09-23 16:54:34.717] [DEBUG]
[2024-09-23 16:54:34.718] [DEBUG] -- 49.52.27.33 port check
[2024-09-23 16:54:34.718] [DEBUG] -- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B47' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:34.822] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.822] [DEBUG] -- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B49' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:34.927] [DEBUG] -- exited code 0
[2024-09-23 16:54:34.957] [DEBUG] - sub start_check ref count to 0
[2024-09-23 16:54:34.957] [DEBUG] - export start_check
[2024-09-23 16:54:34.958] [DEBUG] - Call obagent-py_script_start_check-1.3.0 for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e
[2024-09-23 16:54:34.958] [DEBUG] - import start_check
[2024-09-23 16:54:34.963] [DEBUG] - add start_check ref count to 1
[2024-09-23 16:54:34.964] [INFO] Check before start obagent
[2024-09-23 16:54:34.971] [DEBUG] -- root@49.52.27.33 execute: cat /root/obagent/run/ob_agentd.pid
[2024-09-23 16:54:34.992] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:34.994] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory
[2024-09-23 16:54:34.994] [DEBUG]
[2024-09-23 16:54:34.994] [DEBUG] -- server1(49.52.27.33) port check
[2024-09-23 16:54:34.995] [DEBUG] -- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFD' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:35.099] [DEBUG] -- exited code 0
[2024-09-23 16:54:35.100] [DEBUG] -- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFC' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:35.205] [DEBUG] -- exited code 0
[2024-09-23 16:54:35.206] [DEBUG] -- root@49.52.27.34 execute: cat /root/obagent/run/ob_agentd.pid
[2024-09-23 16:54:35.218] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:35.218] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory
[2024-09-23 16:54:35.219] [DEBUG]
[2024-09-23 16:54:35.219] [DEBUG] -- server2(49.52.27.34) port check
[2024-09-23 16:54:35.220] [DEBUG] -- root@49.52.27.34 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFD' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:35.277] [DEBUG] -- exited code 0
[2024-09-23 16:54:35.277] [DEBUG] -- root@49.52.27.34 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFC' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:35.333] [DEBUG] -- exited code 0
[2024-09-23 16:54:35.333] [DEBUG] -- root@49.52.27.35 execute: cat /root/obagent/run/ob_agentd.pid
[2024-09-23 16:54:35.339] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:35.339] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory
[2024-09-23 16:54:35.339] [DEBUG]
[2024-09-23 16:54:35.340] [DEBUG] -- server3(49.52.27.35) port check
[2024-09-23 16:54:35.340] [DEBUG] -- root@49.52.27.35 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFD' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:35.390] [DEBUG] -- exited code 0
[2024-09-23 16:54:35.391] [DEBUG] -- root@49.52.27.35 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFC' | awk -F' ' '{print $3}' | uniq
[2024-09-23 16:54:35.441] [DEBUG] -- exited code 0
[2024-09-23 16:54:35.492] [DEBUG] - sub start_check ref count to 0
[2024-09-23 16:54:35.492] [DEBUG] - export start_check
[2024-09-23 16:54:35.492] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.0.0 for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-09-23 16:54:35.493] [DEBUG] - import start
[2024-09-23 16:54:35.497] [DEBUG] - add start ref count to 1
[2024-09-23 16:54:35.498] [INFO] Start observer
[2024-09-23 16:54:35.500] [DEBUG] -- root@49.52.27.33 execute: ls /data/1/clog/tenant_1/
[2024-09-23 16:54:35.529] [DEBUG] -- exited code 255, error output:
[2024-09-23 16:54:35.530] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-23 16:54:35.530] [DEBUG]
[2024-09-23 16:54:35.530] [DEBUG] -- root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-23 16:54:35.591] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:35.591] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-23 16:54:35.591] [DEBUG]
[2024-09-23 16:54:35.592] [DEBUG] -- server1(49.52.27.33) start command construction
[2024-09-23 16:54:35.593] [DEBUG] -- root@49.52.27.34 execute: ls /data/1/clog/tenant_1/
[2024-09-23 16:54:35.605] [DEBUG] -- exited code 2, error output:
[2024-09-23 16:54:35.606] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-23 16:54:35.606] [DEBUG]
[2024-09-23 16:54:35.607] [DEBUG] -- root@49.52.27.34 execute: cat /root/observer/run/observer.pid
[2024-09-23 16:54:35.657] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:35.657] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-23 16:54:35.657] [DEBUG]
[2024-09-23 16:54:35.657] [DEBUG] -- server2(49.52.27.34) start command construction
[2024-09-23 16:54:35.658] [DEBUG] -- root@49.52.27.35 execute: ls /data/1/clog/tenant_1/
[2024-09-23 16:54:35.663] [DEBUG] -- exited code 2, error output:
[2024-09-23 16:54:35.664] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-23 16:54:35.664] [DEBUG]
[2024-09-23 16:54:35.665] [DEBUG] -- root@49.52.27.35 execute: cat /root/observer/run/observer.pid
[2024-09-23 16:54:35.710] [DEBUG] -- exited code 1, error output:
[2024-09-23 16:54:35.710] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-23 16:54:35.710] [DEBUG]
[2024-09-23 16:54:35.711] [DEBUG] -- server3(49.52.27.35) start command construction
[2024-09-23 16:54:35.711] [DEBUG] -- starting server1(49.52.27.33) observer
[2024-09-23 16:54:35.712] [DEBUG] -- root@49.52.27.33 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-23 16:54:35.712] [DEBUG] -- root@49.52.27.33 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:2894:2893;49.52.27.34:2894:2893;49.52.27.35:2894:2893' -p 2893 -P 2894 -z 'zone1' -c 1727081654 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='64G',system_memory='30G',datafile_size='80G',log_disk_size='80G'
[2024-09-23 16:54:35.856] [DEBUG] -- exited code 0
[2024-09-23 16:54:35.857] [DEBUG] -- root@49.52.27.33 delete env LD_LIBRARY_PATH
[2024-09-23 16:54:35.857] [DEBUG] -- starting server2(49.52.27.34) observer
[2024-09-23 16:54:35.857] [DEBUG] -- root@49.52.27.34 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-23 16:54:35.858] [DEBUG] -- root@49.52.27.34 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:2894:2893;49.52.27.34:2894:2893;49.52.27.35:2894:2893' -p 2893 -P 2894 -z 'zone2' -c 1727081654 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='64G',system_memory='30G',datafile_size='80G',log_disk_size='80G'
[2024-09-23 16:54:35.946] [DEBUG] -- exited code 0
[2024-09-23 16:54:35.946] [DEBUG] -- root@49.52.27.34 delete env LD_LIBRARY_PATH
[2024-09-23 16:54:35.947] [DEBUG] -- starting server3(49.52.27.35) observer
[2024-09-23 16:54:35.947] [DEBUG] -- root@49.52.27.35 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-23 16:54:35.947] [DEBUG] -- root@49.52.27.35 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:2894:2893;49.52.27.34:2894:2893;49.52.27.35:2894:2893' -p 2893 -P 2894 -z 'zone3' -c 1727081654 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='64G',system_memory='30G',datafile_size='80G',log_disk_size='80G'
[2024-09-23 16:54:36.021] [DEBUG] -- exited code 0
[2024-09-23 16:54:36.022] [DEBUG] -- root@49.52.27.35 delete env LD_LIBRARY_PATH
[2024-09-23 16:54:36.023] [INFO] observer program health check
[2024-09-23 16:54:39.027] [DEBUG] -- server1(49.52.27.33) program health check
[2024-09-23 16:54:39.027] [DEBUG] -- root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-23 16:54:39.049] [DEBUG] -- exited code 0
[2024-09-23 16:54:39.050] [DEBUG] -- root@49.52.27.33 execute: ls /proc/42178
[2024-09-23 16:54:39.119] [DEBUG] -- exited code 255, error output:
[2024-09-23 16:54:39.119] [DEBUG]
[2024-09-23 16:54:39.119] [DEBUG] -- server2(49.52.27.34) program health check
[2024-09-23 16:54:39.120] [DEBUG] -- root@49.52.27.34 execute: cat /root/observer/run/observer.pid
1 个赞
#3 
- sub start_check ref count to 0
- export start_check
- oceanbase-ce starting check failed.
- Call obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.2.0.0-7.el7-b3ead2f667d69fe3195fd22002de83df6eb19382
- import start_check
- add start_check ref count to 1
Check before start obproxy ok
-- root@49.52.27.33 execute: cat /root/obproxy/run/obproxy-49.52.27.33-2887.pid -- exited code 1, error output:
cat: /root/obproxy/run/obproxy-49.52.27.33-2887.pid: No such file or directory

-- 49.52.27.33 port check
-- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B47' | awk -F' ' '{print $3}' | uniq -- exited code 0
-- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B49' | awk -F' ' '{print $3}' | uniq -- exited code 0

- sub start_check ref count to 0
- export start_check
- Call obagent-py_script_start_check-1.3.0 for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e
- import start_check
- add start_check ref count to 1
Check before start obagent ok
-- root@49.52.27.33 execute: cat /root/obagent/run/ob_agentd.pid -- exited code 1, error output:
cat: /root/obagent/run/ob_agentd.pid: No such file or directory

-- server1(49.52.27.33) port check
-- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFD' | awk -F' ' '{print $3}' | uniq -- exited code 0
-- root@49.52.27.33 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFC' | awk -F' ' '{print $3}' | uniq -- exited code 0
-- root@49.52.27.34 execute: cat /root/obagent/run/ob_agentd.pid -- exited code 1, error output:
cat: /root/obagent/run/ob_agentd.pid: No such file or directory

-- server2(49.52.27.34) port check
-- root@49.52.27.34 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFD' | awk -F' ' '{print $3}' | uniq -- exited code 0
-- root@49.52.27.34 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFC' | awk -F' ' '{print $3}' | uniq -- exited code 0
-- root@49.52.27.35 execute: cat /root/obagent/run/ob_agentd.pid -- exited code 1, error output:
cat: /root/obagent/run/ob_agentd.pid: No such file or directory

-- server3(49.52.27.35) port check
-- root@49.52.27.35 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFD' | awk -F' ' '{print $3}' | uniq -- exited code 0
-- root@49.52.27.35 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1FFC' | awk -F' ' '{print $3}' | uniq -- exited code 0

- sub start_check ref count to 0

主要的报错好像就是这些

#4

agent安装了么 ps -ef|grep agent看一下

#5

发下observer.log

#6

截屏2024-09-23 17.55.48
截屏2024-09-23 17.55.481144×116 29 KB

结果是这样的：

#7

log量比较大，请问有比较明确的一些要关注的地方嘛

#8

手动单独启动obd cluster start xxx -c oceanbase-ce

#9

压缩后多大呢？可以附件上传下

#10

截屏2024-09-23 18.14.41
截屏2024-09-23 18.14.41996×418 80.5 KB

#11

observer.log (5.2 MB)
我重新安装了一次，还是一样的报错，麻烦您看看了。这些其他服务器的 error 应该不影响 33 这个机器：

截屏2024-09-23 18.19.59
截屏2024-09-23 18.19.59988×808 143 KB

#12

obd cluster autodeploy obtest -c obtest.yaml -f 我就是这种普通的指令，看起来也没什么，官网说的memory_limit / 3 <= system_memory <= memory_limit / 2也符合，折腾一天很奇怪。

#13

@论坛小助手

#14

根据您提供的日志信息和配置文件内容，OceanBase 在启动过程中遇到了一些问题。具体来说，日志中提到的错误信息表明系统在检查必要的文件和目录时发现它们不存在。这可能是导致 OceanBase 无法启动的原因之一。以下是几个可能的解决方案：

  1. 检查文件和目录是否存在：从日志信息来看，系统尝试访问 /data/1/clog/tenant_1/ 和 /data/1/sstable/block_file 目录，但这些目录似乎不存在。请确保这些目录已经创建，并且具有正确的权限。 [2] 
mkdir -p /data/1/clog/tenant_1
mkdir -p /data/1/sstable/block_file
chmod -R 755 /data/1
chown -R your_username:your_groupname /data/1
  1. 检查配置文件中的路径是否正确：您提供的 YAML 配置文件中指定了 data_dir 和 redo_dir，请确保这些路径与您实际创建的目录一致。 [2] 
oceanbase-ce:
  ...
  global:
    ...
    data_dir: /data/1
    redo_dir: /data/log1
  1. 确认 observer.pid 文件是否存在：日志中还提到尝试读取 /root/observer/run/observer.pid 文件，但该文件不存在。这通常意味着之前没有成功启动过 OceanBase 进程。 [2] 
ls /root/observer/run/observer.pid

如果该文件确实不存在，那么您可以尝试删除 /root/observer/run 目录下的所有文件，然后重新启动 OceanBase。

  1. 检查其他配置项：确保其他配置项如 mysql_port, rpc_port 等没有冲突，并且符合您的需求。 [2] 
oceanbase-ce:
  ...
  global:
    ...
    mysql_port: 2893
    rpc_port: 2894
  1. 确认网络设备名称：您在每个服务器的配置中指定了 devname 为 enp95s0f0，请确保这个设备名称在每台服务器上都是正确的。 [2] 
server1:
  devname: enp95s0f0
  zone: zone1
server2:
  devname: enp95s0f0
  zone: zone2
server3:
  devname: enp95s0f0
  zone: zone3

综上所述，您可以按照以上步骤逐一排查问题，确保所有必要的文件和目录存在，并且配置正确。如果问题仍然存在，建议您联系 OceanBase 的技术支持获取进一步的帮助。

检索到的文档如下:

  1. 4000 ~ 4499
  2. 使用 systemd 部署 OceanBase 数据库
  3. 单机在线转分布式部署
  4. 集群配置项
  5. OceanBase 数据库 TPC-H 测试
  6. 700.7000-7999-of-mysql-mode.md
#15

还有在线人工嘛，感觉还是没有比较明确的解决方案

#16

@Gone222

[2024-09-23 16:54:34.112] [WARNING] (49.52.27.35) failed to get open files
[2024-09-23 16:54:34.112] [WARNING] (49.52.27.35) failed to get max user processes
[2024-09-23 16:54:34.112] [WARNING] OBD-1007: (49.52.27.35) The recommended number of stack size is unlimited (Current value: 10240)

所附上的日志，显示为DEBUG，INFO，WARNING。
按道理均可以忽略。

但是似乎在安装过程中没有按照标准进行配置。

可以考虑先排除这三条WARNING的干扰。

具体如下：

ssh  49.52.27.33/34/45

编辑如下文件，并添加这些行，如果已经有类似的行，记得注释掉（加#号注释），以此为准。
vim /etc/security/limits.conf
*       soft    nofile  655350
*       hard    nofile  655350
*       soft    nproc  655350
*       hard    nproc  655350
*       soft    stack   unlimited
*       hard    stack   unlimited
*       soft    core    unlimited
*       hard    core    unlimited
保存退出，即生效。

同时要注意，如果，有/etc/security/limits.d/20-nproc.conf文件，将其内容全部注释。
（加#号注释）
vim /etc/security/limits.d/20-nproc.conf
# *   ..................
#root ...............
保存退出，即生效。

然后进行启动OB。

#17

我提出此建议的原因在于。

目标服务器似乎在运行多种服务，有PostgreSQL, Ceph, Docker, （Tidb?）,OceanBase。

对于这些服务的运行，可能已经占用了较多的open files和number of process。需要加大此设置。

#18

别睡啊！问题还没解决呢。

2024-09-23 22:18追加如下：

行吧，如果明天有计划且有机会，请尝试一下，因为想知道我的答案是否可以解决你的问题。^_*（我真的不是盯着悬赏来的。）

#19

yaml文件添加一个production_mode: false

#20

还是不行，现在我已经想用一个很简单的配置了，还是有个机器启动不了：

user:
    username: root
    password: 123456
oceanbase-ce:
  version: 4.2.0.0
  servers:
    - name: server1
      ip: 49.52.27.33
    - name: server2
      ip: 49.52.27.34
    - name: server3
      ip: 49.52.27.35
  global:
    devname: enp95s0f0
    memory_limit: 10G # The maximum running memory for an observer
    system_memory: 4G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
    datafile_size: 10G # Size of the data file.
    datafile_next: 10G # the auto extend step. Please enter an capacity, such as 2G
    datafile_maxsize: 20G # the auto extend max size. Please enter an capacity, such as 20G
    log_disk_size: 13G # The size of disk space used by the clog files.
    production_mode: false
    enable_syslog_wf: false
    max_syslog_file_count: 4
    data_dir: /data/1
    redo_dir: /data/log1
  server1:
    zone: zone1
    mysql_port: 3881 
    rpc_port: 3882
    obshell_port: 3886
    home_path: /root/observer
  server2:
    zone: zone2
    mysql_port: 3881 
    rpc_port: 3882
    obshell_port: 3886
    home_path: /root/observer
  server3:
    zone: zone3
    mysql_port: 3881 
    rpc_port: 3882
    obshell_port: 3886
    home_path: /root/observer
obproxy-ce:
  version: 4.2.0.0
  depends:
    - oceanbase-ce
  servers:
    - 49.52.27.33
  global:
    listen_port: 3883
    prometheus_listen_port: 3884
    home_path: /root/obproxy
    enable_cluster_checkout: false
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false

一直显示这个报错：

[2024-09-24 11:29:00.454] [INFO] Start observer
[2024-09-24 11:29:00.454] [DEBUG] -- root@49.52.27.33 execute: ls /data/1/clog/tenant_1/
[2024-09-24 11:29:00.469] [DEBUG] -- exited code 255, error output:
[2024-09-24 11:29:00.470] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-24 11:29:00.470] [DEBUG]
[2024-09-24 11:29:00.470] [DEBUG] -- root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:00.521] [DEBUG] -- exited code 1, error output:
[2024-09-24 11:29:00.521] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-24 11:29:00.521] [DEBUG]
[2024-09-24 11:29:00.521] [DEBUG] -- server1(49.52.27.33) start command construction
[2024-09-24 11:29:00.521] [DEBUG] -- root@49.52.27.34 execute: ls /data/1/clog/tenant_1/
[2024-09-24 11:29:00.533] [DEBUG] -- exited code 2, error output:
[2024-09-24 11:29:00.533] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-24 11:29:00.534] [DEBUG]
[2024-09-24 11:29:00.534] [DEBUG] -- root@49.52.27.34 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:00.585] [DEBUG] -- exited code 1, error output:
[2024-09-24 11:29:00.586] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-24 11:29:00.586] [DEBUG]
[2024-09-24 11:29:00.586] [DEBUG] -- server2(49.52.27.34) start command construction
[2024-09-24 11:29:00.586] [DEBUG] -- root@49.52.27.35 execute: ls /data/1/clog/tenant_1/
[2024-09-24 11:29:00.591] [DEBUG] -- exited code 2, error output:
[2024-09-24 11:29:00.592] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-24 11:29:00.592] [DEBUG]
[2024-09-24 11:29:00.592] [DEBUG] -- root@49.52.27.35 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:00.636] [DEBUG] -- exited code 1, error output:
[2024-09-24 11:29:00.636] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-24 11:29:00.636] [DEBUG]
[2024-09-24 11:29:00.636] [DEBUG] -- server3(49.52.27.35) start command construction
[2024-09-24 11:29:00.637] [DEBUG] -- starting server1(49.52.27.33) observer
[2024-09-24 11:29:00.637] [DEBUG] -- root@49.52.27.33 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-24 11:29:00.637] [DEBUG] -- root@49.52.27.33 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:3882:3881;49.52.27.34:3882:3881;49.52.27.35:3882:3881' -p 3881 -P 3882 -z 'zone1' -c 1727148513 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='10G',system_memory='4G',datafile_size='10G',datafile_next='10G',datafile_maxsize='20G',log_disk_size='13G',enable_syslog_wf=False,max_syslog_file_count=4,obshell_port=3886
[2024-09-24 11:29:00.720] [DEBUG] -- exited code 0
[2024-09-24 11:29:00.721] [DEBUG] -- root@49.52.27.33 delete env LD_LIBRARY_PATH
[2024-09-24 11:29:00.722] [DEBUG] -- starting server2(49.52.27.34) observer
[2024-09-24 11:29:00.722] [DEBUG] -- root@49.52.27.34 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-24 11:29:00.723] [DEBUG] -- root@49.52.27.34 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:3882:3881;49.52.27.34:3882:3881;49.52.27.35:3882:3881' -p 3881 -P 3882 -z 'zone2' -c 1727148513 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='10G',system_memory='4G',datafile_size='10G',datafile_next='10G',datafile_maxsize='20G',log_disk_size='13G',enable_syslog_wf=False,max_syslog_file_count=4,obshell_port=3886
[2024-09-24 11:29:00.806] [DEBUG] -- exited code 0
[2024-09-24 11:29:00.807] [DEBUG] -- root@49.52.27.34 delete env LD_LIBRARY_PATH
[2024-09-24 11:29:00.807] [DEBUG] -- starting server3(49.52.27.35) observer
[2024-09-24 11:29:00.807] [DEBUG] -- root@49.52.27.35 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-24 11:29:00.808] [DEBUG] -- root@49.52.27.35 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:3882:3881;49.52.27.34:3882:3881;49.52.27.35:3882:3881' -p 3881 -P 3882 -z 'zone3' -c 1727148513 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='10G',system_memory='4G',datafile_size='10G',datafile_next='10G',datafile_maxsize='20G',log_disk_size='13G',enable_syslog_wf=False,max_syslog_file_count=4,obshell_port=3886
[2024-09-24 11:29:00.881] [DEBUG] -- exited code 0
[2024-09-24 11:29:00.882] [DEBUG] -- root@49.52.27.35 delete env LD_LIBRARY_PATH
[2024-09-24 11:29:00.977] [INFO] observer program health check
[2024-09-24 11:29:03.980] [DEBUG] -- server1(49.52.27.33) program health check
[2024-09-24 11:29:03.980] [DEBUG] -- root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:03.993] [DEBUG] -- exited code 0
[2024-09-24 11:29:03.994] [DEBUG] -- root@49.52.27.33 execute: ls /proc/12246
[2024-09-24 11:29:04.049] [DEBUG] -- exited code 255, error output:
[2024-09-24 11:29:04.049] [DEBUG]
[2024-09-24 11:29:04.050] [DEBUG] -- server2(49.52.27.34) program health check
[2024-09-24 11:29:04.050] [DEBUG] -- root@49.52.27.34 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:04.062] [DEBUG] -- exited code 0
[2024-09-24 11:29:04.063] [DEBUG] -- root@49.52.27.34 execute: ls /proc/2514
[2024-09-24 11:29:04.114] [DEBUG] -- exited code 0
[2024-09-24 11:29:04.115] [DEBUG] -- server2(49.52.27.34) observer[pid: 2514] started
[2024-09-24 11:29:04.115] [DEBUG] -- server3(49.52.27.35) program health check
[2024-09-24 11:29:04.115] [DEBUG] -- root@49.52.27.35 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:04.121] [DEBUG] -- exited code 0
[2024-09-24 11:29:04.122] [DEBUG] -- root@49.52.27.35 execute: ls /proc/105710
[2024-09-24 11:29:04.167] [DEBUG] -- exited code 0
[2024-09-24 11:29:04.167] [DEBUG] -- server3(49.52.27.35) observer[pid: 105710] started
[2024-09-24 11:29:04.235] [WARNING] OBD-2002: Failed to start server1(49.52.27.33) observer
[2024-09-24 11:29:04.235] [DEBUG] - sub start ref count to 0
[2024-09-24 11:29:04.235] [DEBUG] - export start
[2024-09-24 11:29:04.235] [ERROR] oceanbase-ce start failed
[2024-09-24 11:29:04.239] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2024-09-24 11:29:04.239] [INFO] Trace ID: 21595dec-7a25-11ef-8104-b008754a7b77
[2024-09-24 11:29:04.239] [INFO] If you want to view detailed obd logs, please run: obd display-trace 21595dec-7a25-11ef-8104-b008754a7b77
[2024-09-24 11:29:04.240] [DEBUG] - share lock /home/xjk/.obd/lock/mirror_and_repo release, count 1
[2024-09-24 11:29:04.240] [DEBUG] - share lock /home/xjk/.obd/lock/mirror_and_repo release, count 0
[2024-09-24 11:29:04.240] [DEBUG] - unlock /home/xjk/.obd/lock/mirror_and_repo
[2024-09-24 11:29:04.240] [DEBUG] - exclusive lock /home/xjk/.obd/lock/deploy_obtest release, count 0
[2024-09-24 11:29:04.240] [DEBUG] - unlock /home/xjk/.obd/lock/deploy_obtest
[2024-09-24 11:29:04.240] [DEBUG] - share lock /home/xjk/.obd/lock/global release, count 0
[2024-09-24 11:29:04.240] [DEBUG] - unlock /home/xjk/.obd/lock/global
#21

但是我是 33 机器启动不起来，看 ob 给我的报错都是 warn 在另外两个节点的