OceanBase启动报错 failed to connect meta db ocp-express start failed observer need bootstarp

OceanBase 技术问题
#1

【 使用环境 】生产环境
【 OB or 其他组件 】ob and ocp e
【 使用版本 】4.1.0
【问题描述】
OceanBase启动报错 failed to connect meta db ocp-express start failed observer need bootstarp
【复现路径】问题出现前后相关操作
【问题现象及影响】

[oceanbase@localhost ~]$ obd cluster start fund
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Check before start observer ok
[WARN] OBD-1011: (10.0.10.207) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[WARN] OBD-1007: (10.0.10.207) The recommended number of open files is 655350 (Current value: 20000)

Check before start obproxy ok
Check before start obagent ok
Check before start ocp-express ok
Start observer ok
observer program health check ok
Connect to observer ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Initialize obproxy-ce ok
Start obagent ok
obagent program health check ok
Connect to Obagent ok
Start ocp-express x
[ERROR] 10.0.10.207: failed to connect meta db

[ERROR] ocp-express start failed
observer need bootstarp x
±----------------------------------------------+
| obproxy |
±------------±-----±----------------±-------+
| ip | port | prometheus_port | status |
±------------±-----±----------------±-------+
| 10.0.10.207 | 2883 | 2884 | active |
±------------±-----±----------------±-------+
obclient -h10.0.10.207 -P2883 -uroot -p’AbcD123!@#’ -Doceanbase -A
±---------------------------------------------------------------+
| obagent |
±------------±-------------------±-------------------±-------+
| ip | mgragent_http_port | monagent_http_port | status |
±------------±-------------------±-------------------±-------+
| 10.0.10.207 | 8089 | 8088 | active |
±------------±-------------------±-------------------±-------+
See OceanBase分布式数据库-海量数据 笔笔算数 .
Trace ID: 3b9e03c8-d9a5-11ed-a61c-00155d1f7e00
If you want to view detailed obd logs, please run: obd display-trace 3b9e03c8-d9a5-11ed-a61c-00155d1f7e00
[oceanbase@localhost ~]$ ^C
[oceanbase@localhost ~]$ obd cluster stop fund
Get local repositories ok
Search plugins ok
Open ssh connection ok
Stop observer ok
Stop obproxy ok
Stop obagent ok
Stop ocp-express ok
fund stopped
Trace ID: d56370a6-d9a5-11ed-b17c-00155d1f7e00
If you want to view detailed obd logs, please run: obd display-trace d56370a6-d9a5-11ed-b17c-00155d1f7e00
[oceanbase@localhost ~]$ obd cluster start fund
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Check before start observer ok
[WARN] OBD-1011: (10.0.10.207) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[WARN] OBD-1007: (10.0.10.207) The recommended number of open files is 655350 (Current value: 20000)

Check before start obproxy ok
Check before start obagent ok
Check before start ocp-express ok
Start observer ok
observer program health check ok
Connect to observer ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Initialize obproxy-ce ok
Start obagent ok
obagent program health check ok
Connect to Obagent ok
Start ocp-express x
[ERROR] 10.0.10.207: failed to connect meta db

[ERROR] ocp-express start failed
observer need bootstarp x
±----------------------------------------------+
| obproxy |
±------------±-----±----------------±-------+
| ip | port | prometheus_port | status |
±------------±-----±----------------±-------+
| 10.0.10.207 | 2883 | 2884 | active |
±------------±-----±----------------±-------+
obclient -h10.0.10.207 -P2883 -uroot -p’AbcD123!@#’ -Doceanbase -A
±---------------------------------------------------------------+
| obagent |
±------------±-------------------±-------------------±-------+
| ip | mgragent_http_port | monagent_http_port | status |
±------------±-------------------±-------------------±-------+
| 10.0.10.207 | 8089 | 8088 | active |
±------------±-------------------±-------------------±-------+
See OceanBase分布式数据库-海量数据 笔笔算数 .
Trace ID: f5a2356e-d9a5-11ed-90a2-00155d1f7e00
If you want to view detailed obd logs, please run: obd display-trace f5a2356e-d9a5-11ed-90a2-00155d1f7e00

【附件】

#3

这是通过 obd web 安装的吗？看起来是ocp express的元数据库初始化失败，有 observer.log 看看吗

#4

[oceanbase@localhost ~]$ obd display-trace f5a2356e-d9a5-11ed-90a2-00155d1f7e00
[2023-04-13 10:50:33.223] [DEBUG] - mkdir /home/oceanbase/.obd/lock/
[2023-04-13 10:50:33.224] [DEBUG] - unknown lock mode
[2023-04-13 10:50:33.224] [DEBUG] - try to get share lock /home/oceanbase/.obd/lock/global
[2023-04-13 10:50:33.224] [DEBUG] - share lock /home/oceanbase/.obd/lock/global, count 1
[2023-04-13 10:50:33.224] [DEBUG] - cmd: [‘fund’]
[2023-04-13 10:50:33.224] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2023-04-13 10:50:33.224] [DEBUG] - Get Deploy by name
[2023-04-13 10:50:33.224] [DEBUG] - mkdir /home/oceanbase/.obd/cluster/
[2023-04-13 10:50:33.224] [DEBUG] - mkdir /home/oceanbase/.obd/config_parser/
[2023-04-13 10:50:33.225] [DEBUG] - try to get exclusive lock /home/oceanbase/.obd/lock/deploy_fund
[2023-04-13 10:50:33.225] [DEBUG] - exclusive lock /home/oceanbase/.obd/lock/deploy_fund, count 1
[2023-04-13 10:50:33.232] [DEBUG] - Deploy status judge
[2023-04-13 10:50:33.233] [INFO] Get local repositories
[2023-04-13 10:50:33.233] [DEBUG] - mkdir /home/oceanbase/.obd/repository
[2023-04-13 10:50:33.234] [DEBUG] - Get local repository oceanbase-ce-4.1.0.0-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:33.234] [DEBUG] - Search repository oceanbase-ce version: 4.1.0.0, tag: 1fcbef263fe849d1876d9d6f65c45208d0b29844, release: None, package_hash: None
[2023-04-13 10:50:33.234] [DEBUG] - try to get share lock /home/oceanbase/.obd/lock/mirror_and_repo
[2023-04-13 10:50:33.234] [DEBUG] - share lock /home/oceanbase/.obd/lock/mirror_and_repo, count 1
[2023-04-13 10:50:33.234] [DEBUG] - mkdir /home/oceanbase/.obd/repository/oceanbase-ce
[2023-04-13 10:50:33.236] [DEBUG] - Found repository oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:33.236] [DEBUG] - Get local repository obproxy-ce-4.1.0.0-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:33.236] [DEBUG] - Search repository obproxy-ce version: 4.1.0.0, tag: b7751f64893bb2f66a018afeeb7276865fc88f30, release: None, package_hash: None
[2023-04-13 10:50:33.236] [DEBUG] - share lock /home/oceanbase/.obd/lock/mirror_and_repo, count 2
[2023-04-13 10:50:33.237] [DEBUG] - mkdir /home/oceanbase/.obd/repository/obproxy-ce
[2023-04-13 10:50:33.239] [DEBUG] - Found repository obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:33.239] [DEBUG] - Get local repository obagent-1.3.0-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:33.239] [DEBUG] - Search repository obagent version: 1.3.0, tag: 647924a34ac989ce17b9e2d58e942ce3268b3b8e, release: None, package_hash: None
[2023-04-13 10:50:33.239] [DEBUG] - share lock /home/oceanbase/.obd/lock/mirror_and_repo, count 3
[2023-04-13 10:50:33.239] [DEBUG] - mkdir /home/oceanbase/.obd/repository/obagent
[2023-04-13 10:50:33.241] [DEBUG] - Found repository obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:33.241] [DEBUG] - Get local repository ocp-express-1.0.0-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:33.241] [DEBUG] - Search repository ocp-express version: 1.0.0, tag: 093d48d4339920b567f02a7ada83891c86deaf65, release: None, package_hash: None
[2023-04-13 10:50:33.241] [DEBUG] - share lock /home/oceanbase/.obd/lock/mirror_and_repo, count 4
[2023-04-13 10:50:33.241] [DEBUG] - mkdir /home/oceanbase/.obd/repository/ocp-express
[2023-04-13 10:50:33.243] [DEBUG] - Found repository ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:33.364] [DEBUG] - Get deploy config
[2023-04-13 10:50:33.388] [INFO] Search plugins
[2023-04-13 10:50:33.388] [DEBUG] - Searching start_check plugin for components …
[2023-04-13 10:50:33.388] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:33.388] [DEBUG] - mkdir /home/oceanbase/.obd/plugins
[2023-04-13 10:50:33.389] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-04-13 10:50:33.389] [DEBUG] - Searching start_check plugin for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:33.390] [DEBUG] - Found for obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.1.0.0
[2023-04-13 10:50:33.390] [DEBUG] - Searching start_check plugin for obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:33.390] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-1.3.0
[2023-04-13 10:50:33.390] [DEBUG] - Searching start_check plugin for ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:33.390] [DEBUG] - Found for ocp-express-py_script_start_check-1.0 for ocp-express-1.0.0
[2023-04-13 10:50:33.390] [DEBUG] - Searching create_tenant plugin for components …
[2023-04-13 10:50:33.390] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:33.391] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-04-13 10:50:33.391] [DEBUG] - Searching create_tenant plugin for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:33.391] [DEBUG] - No such create_tenant plugin for obproxy-ce-4.1.0.0
[2023-04-13 10:50:33.391] [DEBUG] - Searching create_tenant plugin for obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:33.391] [DEBUG] - No such create_tenant plugin for obagent-1.3.0
[2023-04-13 10:50:33.391] [DEBUG] - Searching create_tenant plugin for ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:33.392] [DEBUG] - No such create_tenant plugin for ocp-express-1.0.0
[2023-04-13 10:50:33.392] [DEBUG] - Searching start plugin for components …
[2023-04-13 10:50:33.392] [DEBUG] - Searching start plugin for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:33.392] [DEBUG] - Found for oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-04-13 10:50:33.392] [DEBUG] - Searching start plugin for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:33.392] [DEBUG] - Found for obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.1.0.0
[2023-04-13 10:50:33.392] [DEBUG] - Searching start plugin for obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:33.393] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-1.3.0
[2023-04-13 10:50:33.393] [DEBUG] - Searching start plugin for ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:33.393] [DEBUG] - Found for ocp-express-py_script_start-1.0 for ocp-express-1.0.0
[2023-04-13 10:50:33.393] [DEBUG] - Searching connect plugin for components …
[2023-04-13 10:50:33.393] [DEBUG] - Searching connect plugin for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:33.394] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.0
[2023-04-13 10:50:33.394] [DEBUG] - Searching connect plugin for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:33.394] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.1.0.0
[2023-04-13 10:50:33.394] [DEBUG] - Searching connect plugin for obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:33.394] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-1.3.0
[2023-04-13 10:50:33.394] [DEBUG] - Searching connect plugin for ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:33.394] [DEBUG] - Found for ocp-express-py_script_connect-1.0 for ocp-express-1.0.0
[2023-04-13 10:50:33.394] [DEBUG] - Searching bootstrap plugin for components …
[2023-04-13 10:50:33.395] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:33.395] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-04-13 10:50:33.395] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:33.395] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.1.0.0
[2023-04-13 10:50:33.395] [DEBUG] - Searching bootstrap plugin for obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:33.396] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-1.3.0
[2023-04-13 10:50:33.396] [DEBUG] - Searching bootstrap plugin for ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:33.396] [DEBUG] - Found for ocp-express-py_script_bootstrap-1.0 for ocp-express-1.0.0
[2023-04-13 10:50:33.396] [DEBUG] - Searching display plugin for components …
[2023-04-13 10:50:33.396] [DEBUG] - Searching display plugin for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:33.396] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.1.0.0
[2023-04-13 10:50:33.397] [DEBUG] - Searching display plugin for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:33.397] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.1.0.0
[2023-04-13 10:50:33.397] [DEBUG] - Searching display plugin for obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:33.397] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-1.3.0
[2023-04-13 10:50:33.397] [DEBUG] - Searching display plugin for ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:33.397] [DEBUG] - Found for ocp-express-py_script_display-1.0 for ocp-express-1.0.0
[2023-04-13 10:50:33.519] [INFO] Open ssh connection
[2023-04-13 10:50:33.651] [INFO] Load cluster param plugin
[2023-04-13 10:50:33.651] [DEBUG] - Searching param plugin for components …
[2023-04-13 10:50:33.651] [DEBUG] - Search param plugin for oceanbase-ce
[2023-04-13 10:50:33.652] [DEBUG] - Found for oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-04-13 10:50:33.652] [DEBUG] - Applying oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:34.249] [DEBUG] - Search param plugin for obproxy-ce
[2023-04-13 10:50:34.249] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.1.0.0
[2023-04-13 10:50:34.249] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:34.390] [DEBUG] - Search param plugin for obagent
[2023-04-13 10:50:34.391] [DEBUG] - Found for obagent-param-1.3.0 for obagent-1.3.0
[2023-04-13 10:50:34.391] [DEBUG] - Applying obagent-param-1.3.0 for obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:34.459] [DEBUG] - Search param plugin for ocp-express
[2023-04-13 10:50:34.460] [DEBUG] - Found for ocp-express-param-1.0 for ocp-express-1.0.0
[2023-04-13 10:50:34.460] [DEBUG] - Applying ocp-express-param-1.0 for ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:34.623] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:34.624] [DEBUG] - import start_check
[2023-04-13 10:50:34.630] [DEBUG] - add start_check ref count to 1
[2023-04-13 10:50:34.630] [INFO] Check before start observer
[2023-04-13 10:50:34.632] [DEBUG] – oceanbase@10.0.10.207 execute: ls /home/oceanbase/fund/oceanbase/store/clog/tenant_1/
[2023-04-13 10:50:34.655] [DEBUG] – exited code 0
[2023-04-13 10:50:34.655] [DEBUG] – oceanbase@10.0.10.207 execute: cat /home/oceanbase/fund/oceanbase/run/observer.pid
[2023-04-13 10:50:34.717] [DEBUG] – exited code 1, error output:
[2023-04-13 10:50:34.717] [DEBUG] cat: /home/oceanbase/fund/oceanbase/run/observer.pid: 没有那个文件或目录
[2023-04-13 10:50:34.718] [DEBUG]
[2023-04-13 10:50:34.718] [DEBUG] – 10.0.10.207 port check
[2023-04-13 10:50:34.718] [DEBUG] – oceanbase@10.0.10.207 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-04-13 10:50:34.786] [DEBUG] – exited code 0
[2023-04-13 10:50:34.786] [DEBUG] – oceanbase@10.0.10.207 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-04-13 10:50:34.858] [DEBUG] – exited code 0
[2023-04-13 10:50:34.859] [DEBUG] – oceanbase@10.0.10.207 execute: ls /home/oceanbase/fund/oceanbase/store/sstable/block_file
[2023-04-13 10:50:34.919] [DEBUG] – exited code 0
[2023-04-13 10:50:34.920] [DEBUG] – oceanbase@10.0.10.207 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2023-04-13 10:50:34.981] [DEBUG] – exited code 0
[2023-04-13 10:50:34.981] [WARNING] OBD-1011: (10.0.10.207) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[2023-04-13 10:50:34.982] [DEBUG] – oceanbase@10.0.10.207 execute: ulimit -a
[2023-04-13 10:50:35.041] [DEBUG] – exited code 0
[2023-04-13 10:50:35.042] [WARNING] OBD-1007: (10.0.10.207) The recommended number of open files is 655350 (Current value: 20000)
[2023-04-13 10:50:35.042] [DEBUG] – oceanbase@10.0.10.207 execute: cat /proc/meminfo
[2023-04-13 10:50:35.103] [DEBUG] – exited code 0
[2023-04-13 10:50:35.104] [DEBUG] – oceanbase@10.0.10.207 execute: df --block-size=1024
[2023-04-13 10:50:35.167] [DEBUG] – exited code 0
[2023-04-13 10:50:35.168] [DEBUG] – get disk info for path /dev, total: 49319256064 avail: 49319256064
[2023-04-13 10:50:35.168] [DEBUG] – get disk info for path /dev/shm, total: 49337491456 avail: 49337491456
[2023-04-13 10:50:35.168] [DEBUG] – get disk info for path /run, total: 49337491456 avail: 49328402432
[2023-04-13 10:50:35.168] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 49337491456 avail: 49337491456
[2023-04-13 10:50:35.168] [DEBUG] – get disk info for path /, total: 75125227520 avail: 71957241856
[2023-04-13 10:50:35.168] [DEBUG] – get disk info for path /boot, total: 1063256064 avail: 860688384
[2023-04-13 10:50:35.168] [DEBUG] – get disk info for path /home, total: 980505796608 avail: 323890155520
[2023-04-13 10:50:35.169] [DEBUG] – get disk info for path /run/user/1000, total: 9867497472 avail: 9867497472
[2023-04-13 10:50:35.169] [DEBUG] – disk: {’/dev’: {‘total’: 49319256064, ‘avail’: 49319256064, ‘need’: 0}, ‘/dev/shm’: {‘total’: 49337491456, ‘avail’: 49337491456, ‘need’: 0}, ‘/run’: {‘total’: 49337491456, ‘avail’: 49328402432, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 49337491456, ‘avail’: 49337491456, ‘need’: 0}, ‘/’: {‘total’: 75125227520, ‘avail’: 71957241856, ‘need’: 0}, ‘/boot’: {‘total’: 1063256064, ‘avail’: 860688384, ‘need’: 0}, ‘/home’: {‘total’: 980505796608, ‘avail’: 323890155520, ‘need’: 0}, ‘/run/user/1000’: {‘total’: 9867497472, ‘avail’: 9867497472, ‘need’: 0}}
[2023-04-13 10:50:35.169] [DEBUG] – oceanbase@10.0.10.207 execute: date +%s%N
[2023-04-13 10:50:35.227] [DEBUG] – exited code 0
[2023-04-13 10:50:35.228] [DEBUG] – 10.0.10.207 time delta -1.177978515625
[2023-04-13 10:50:35.284] [INFO] [WARN] OBD-1011: (10.0.10.207) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[2023-04-13 10:50:35.284] [INFO] [WARN] OBD-1007: (10.0.10.207) The recommended number of open files is 655350 (Current value: 20000)
[2023-04-13 10:50:35.284] [INFO]
[2023-04-13 10:50:35.284] [DEBUG] - sub start_check ref count to 0
[2023-04-13 10:50:35.284] [DEBUG] - export start_check
[2023-04-13 10:50:35.284] [DEBUG] - Call obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:35.284] [DEBUG] - import start_check
[2023-04-13 10:50:35.286] [DEBUG] - add start_check ref count to 1
[2023-04-13 10:50:35.287] [INFO] Check before start obproxy
[2023-04-13 10:50:35.287] [DEBUG] – oceanbase@10.0.10.207 execute: cat /home/oceanbase/fund/obproxy/run/obproxy-10.0.10.207-2883.pid
[2023-04-13 10:50:35.311] [DEBUG] – exited code 1, error output:
[2023-04-13 10:50:35.311] [DEBUG] cat: /home/oceanbase/fund/obproxy/run/obproxy-10.0.10.207-2883.pid: 没有那个文件或目录
[2023-04-13 10:50:35.311] [DEBUG]
[2023-04-13 10:50:35.312] [DEBUG] – 10.0.10.207 port check
[2023-04-13 10:50:35.312] [DEBUG] – oceanbase@10.0.10.207 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B43’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-04-13 10:50:35.382] [DEBUG] – exited code 0
[2023-04-13 10:50:35.382] [DEBUG] – oceanbase@10.0.10.207 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B44’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-04-13 10:50:35.451] [DEBUG] – exited code 0
[2023-04-13 10:50:35.548] [DEBUG] - sub start_check ref count to 0
[2023-04-13 10:50:35.549] [DEBUG] - export start_check
[2023-04-13 10:50:35.549] [DEBUG] - Call obagent-py_script_start_check-1.3.0 for obagent-1.3.0-22.el8-647924a34ac989ce17b9e2d58e942ce3268b3b8e
[2023-04-13 10:50:35.549] [DEBUG] - import start_check
[2023-04-13 10:50:35.551] [DEBUG] - add start_check ref count to 1
[2023-04-13 10:50:35.551] [INFO] Check before start obagent
[2023-04-13 10:50:35.553] [DEBUG] – oceanbase@10.0.10.207 execute: cat /home/oceanbase/fund/obagent/run/ob_agentd.pid
[2023-04-13 10:50:35.575] [DEBUG] – exited code 1, error output:
[2023-04-13 10:50:35.575] [DEBUG] cat: /home/oceanbase/fund/obagent/run/ob_agentd.pid: 没有那个文件或目录
[2023-04-13 10:50:35.575] [DEBUG]
[2023-04-13 10:50:35.575] [DEBUG] – 10.0.10.207 port check
[2023-04-13 10:50:35.575] [DEBUG] – oceanbase@10.0.10.207 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:1F99’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-04-13 10:50:35.645] [DEBUG] – exited code 0
[2023-04-13 10:50:35.645] [DEBUG] – oceanbase@10.0.10.207 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:1F98’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-04-13 10:50:35.715] [DEBUG] – exited code 0
[2023-04-13 10:50:35.814] [DEBUG] - sub start_check ref count to 0
[2023-04-13 10:50:35.815] [DEBUG] - export start_check
[2023-04-13 10:50:35.815] [DEBUG] - Call ocp-express-py_script_start_check-1.0 for ocp-express-1.0.0-100000432023032015.el8-093d48d4339920b567f02a7ada83891c86deaf65
[2023-04-13 10:50:35.815] [DEBUG] - import start_check
[2023-04-13 10:50:35.820] [DEBUG] - add start_check ref count to 1
[2023-04-13 10:50:35.820] [INFO] Check before start ocp-express
[2023-04-13 10:50:35.823] [DEBUG] – oceanbase@10.0.10.207 execute: cat /home/oceanbase/fund/ocpexpress/run/ocp-express.pid
[2023-04-13 10:50:35.844] [DEBUG] – exited code 0
[2023-04-13 10:50:35.844] [DEBUG] – oceanbase@10.0.10.207 execute: ls /proc/9849
[2023-04-13 10:50:35.906] [DEBUG] – exited code 2, error output:
[2023-04-13 10:50:35.906] [DEBUG] ls: 无法访问’/proc/9849’: 没有那个文件或目录
[2023-04-13 10:50:35.906] [DEBUG]
[2023-04-13 10:50:35.906] [DEBUG] – oceanbase@10.0.10.207 execute: bash -c ‘cat /proc/net/{udp*,tcp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:1FF4’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-04-13 10:50:35.977] [DEBUG] – exited code 0
[2023-04-13 10:50:35.978] [DEBUG] – oceanbase@10.0.10.207 execute: java -version
[2023-04-13 10:50:36.104] [DEBUG] – exited code 0
[2023-04-13 10:50:36.104] [DEBUG] – oceanbase@10.0.10.207 execute: cat /proc/meminfo
[2023-04-13 10:50:36.165] [DEBUG] – exited code 0
[2023-04-13 10:50:36.165] [DEBUG] – oceanbase@10.0.10.207 execute: df --block-size=1024
[2023-04-13 10:50:36.231] [DEBUG] – exited code 0
[2023-04-13 10:50:36.231] [DEBUG] – get disk info for path /dev, total: 49319256064 avail: 49319256064
[2023-04-13 10:50:36.231] [DEBUG] – get disk info for path /dev/shm, total: 49337491456 avail: 49337491456
[2023-04-13 10:50:36.231] [DEBUG] – get disk info for path /run, total: 49337491456 avail: 49328402432
[2023-04-13 10:50:36.231] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 49337491456 avail: 49337491456
[2023-04-13 10:50:36.231] [DEBUG] – get disk info for path /, total: 75125227520 avail: 71957241856
[2023-04-13 10:50:36.231] [DEBUG] – get disk info for path /boot, total: 1063256064 avail: 860688384
[2023-04-13 10:50:36.232] [DEBUG] – get disk info for path /home, total: 980505796608 avail: 323890155520
[2023-04-13 10:50:36.232] [DEBUG] – get disk info for path /run/user/1000, total: 9867497472 avail: 9867497472
[2023-04-13 10:50:36.232] [DEBUG] – oceanbase@10.0.10.207 execute: df --block-size=1024 /home/oceanbase/fund/ocpexpress/log
[2023-04-13 10:50:36.293] [DEBUG] – exited code 0
[2023-04-13 10:50:36.294] [DEBUG] – get disk info for path /home, total: 980505796608 avail: 323890155520
[2023-04-13 10:50:36.342] [DEBUG] - sub start_check ref count to 0
[2023-04-13 10:50:36.343] [DEBUG] - export start_check
[2023-04-13 10:50:36.343] [DEBUG] - Call oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:36.343] [DEBUG] - import start
[2023-04-13 10:50:36.345] [DEBUG] - add start ref count to 1
[2023-04-13 10:50:36.345] [INFO] Start observer
[2023-04-13 10:50:36.345] [DEBUG] – oceanbase@10.0.10.207 execute: ls /home/oceanbase/fund/oceanbase/store/clog/tenant_1/
[2023-04-13 10:50:36.367] [DEBUG] – exited code 0
[2023-04-13 10:50:36.368] [DEBUG] – oceanbase@10.0.10.207 execute: cat /home/oceanbase/fund/oceanbase/run/observer.pid
[2023-04-13 10:50:36.431] [DEBUG] – exited code 1, error output:
[2023-04-13 10:50:36.431] [DEBUG] cat: /home/oceanbase/fund/oceanbase/run/observer.pid: 没有那个文件或目录
[2023-04-13 10:50:36.431] [DEBUG]
[2023-04-13 10:50:36.431] [DEBUG] – 10.0.10.207 start command construction
[2023-04-13 10:50:36.432] [DEBUG] – starting 10.0.10.207 observer
[2023-04-13 10:50:36.432] [DEBUG] – oceanbase@10.0.10.207 set env LD_LIBRARY_PATH to ‘/home/oceanbase/fund/oceanbase/lib:’
[2023-04-13 10:50:36.432] [DEBUG] – oceanbase@10.0.10.207 execute: cd /home/oceanbase/fund/oceanbase; /home/oceanbase/fund/oceanbase/bin/observer -r ‘10.0.10.207:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -n ‘fund’ -c 1 -d ‘/home/oceanbase/fund/oceanbase/store’ -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit=‘81G’,datafile_size=‘500G’,datafile_disk_percentage=50,data_disk_usage_limit_percentage=70,system_memory=‘18G’,log_disk_size=‘100G’,log_disk_percentage=50,log_disk_utilization_threshold=50,cpu_count=46
[2023-04-13 10:50:36.542] [DEBUG] – exited code 0
[2023-04-13 10:50:36.543] [DEBUG] – oceanbase@10.0.10.207 delete env LD_LIBRARY_PATH
[2023-04-13 10:50:36.607] [INFO] observer program health check
[2023-04-13 10:50:39.609] [DEBUG] – 10.0.10.207 program health check
[2023-04-13 10:50:39.609] [DEBUG] – oceanbase@10.0.10.207 execute: cat /home/oceanbase/fund/oceanbase/run/observer.pid
[2023-04-13 10:50:39.631] [DEBUG] – exited code 0
[2023-04-13 10:50:39.631] [DEBUG] – oceanbase@10.0.10.207 execute: ls /proc/55328
[2023-04-13 10:50:39.698] [DEBUG] – exited code 0
[2023-04-13 10:50:39.699] [DEBUG] – 10.0.10.207 observer[pid: 55328] started
[2023-04-13 10:50:39.737] [DEBUG] - sub start ref count to 0
[2023-04-13 10:50:39.737] [DEBUG] - export start
[2023-04-13 10:50:39.738] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.0-100000192023032010.el8-1fcbef263fe849d1876d9d6f65c45208d0b29844
[2023-04-13 10:50:39.738] [DEBUG] - import connect
[2023-04-13 10:50:39.769] [DEBUG] - add connect ref count to 1
[2023-04-13 10:50:39.770] [INFO] Connect to observer
[2023-04-13 10:50:39.771] [DEBUG] – connect 10.0.10.207 -P2881 -uroot -pAbcD123!@#
[2023-04-13 10:50:42.777] [DEBUG] – connect 10.0.10.207 -P2881 -uroot -p
[2023-04-13 10:50:43.029] [DEBUG] - sub connect ref count to 0
[2023-04-13 10:50:43.030] [DEBUG] - export connect
[2023-04-13 10:50:43.030] [DEBUG] - Call obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.1.0.0-7.el8-b7751f64893bb2f66a018afeeb7276865fc88f30
[2023-04-13 10:50:43.030] [DEBUG] - import start
[2023-04-13 10:50:43.032] [DEBUG] - add start ref count to 1
[2023-04-13 10:50:43.034] [INFO] Start obproxy
[2023-04-13 10:50:43.035] [DEBUG] – oceanbase@10.0.10.207 execute: ls /home/oceanbase/fund/obproxy/etc/obproxy_config.bin
[2023-04-13 10:50:43.058] [DEBUG] – exited code 0
[2023-04-13 10:50:43.058] [DEBUG] – 10.0.10.207 port check
[2023-04-13 10:50:43.059] [DEBUG] – oceanbase@10.0.10.207 execute: cat /home/oceanbase/fund/obproxy/run/obproxy-10.0.10.207-2883.pid
[2023-04-13 10:50:43.130] [DEBUG] – exited code 1, error output:
[2023-04-13 10:50:43.131] [DEBUG] cat: /home/oceanbase/fund/obproxy/run/obproxy-10.0.10.207-2883.pid: 没有那个文件或目录
[2023-04-13 10:50:43.131] [DEBUG]
[2023-04-13 10:50:43.131] [DEBUG] – starting 10.0.10.207 obproxy
[2023-04-13 10:50:43.131] [DEBUG] – oceanbase@10.0.10.207 set env LD_LIBRARY_PATH to ‘/home/oceanbase/fund/obproxy/lib:’
[2023-04-13 10:50:43.131] [DEBUG] – oceanbase@10.0.10.207 execute: cd /home/oceanbase/fund/obproxy; /home/oceanbase/fund/obproxy/bin/obproxy -o obproxy_sys_password=’’,enable_strict_kernel_release=False,skip_proxy_sys_private_check=True,enable_cluster_checkout=False --listen_port 2883 --prometheus_listen_port 2884 --rs_list ‘10.0.10.207:2881’ --cluster_name ‘fund’
[2023-04-13 10:50:43.210] [DEBUG] – exited code 0
[2023-04-13 10:50:43.210] [DEBUG] – oceanbase@10.0.10.207 delete env LD_LIBRARY_PATH
[2023-04-13 10:50:43.210] [DEBUG] – oceanbase@10.0.10.207 execute: ps -aux | grep -e ‘/home/oceanbase/fund/obproxy/bin/obproxy -o obproxy_sys_password=,enable_strict_kernel_release=False,skip_proxy_sys_private_check=True,enable_cluster_checkout=False --listen_port 2883 --prometheus_listen_port 2884 --rs_list 10.0.10.207:2881 --cluster_name fund$’ | grep -v grep | awk ‘{print $2}’ > /home/oceanbase/fund/obproxy/run/obproxy-10.0.10.207-2883.pid
[2023-04-13 10:50:43.311] [DEBUG] – exited code 0
[2023-04-13 10:50:43.427] [INFO] obproxy program health check
[2023-04-13 10:50:43.427] [DEBUG] – 10.0.10.207 program health check
[2023-04-13 10:50:43.427] [DEBUG] – oceanbase@10.0.10.207 execute: cat /home/oceanbase/fund/obproxy/run/obproxy-10.0.10.207-2883.pid

#5

你这个是 obd 的日志，observer 的进程是不是已经起来了？发一下 observer.log 看看？

#6

observer.zip (2.8 MB)

#7

是通过 obd web 安装的吗？

#8

是的， obd web 安装。 谢谢！

#9

[2023-04-13 10:50:48.619056] ERROR start (ob_server.cpp:896) [55328][][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4393] observer start process failure(msg=“observer start() has failure”, ret=-4624, ret=“OB_MACHINE_RESOURCE_NOT_ENOUGH”)

不知道是什么资源不够 :grinning:

#11

image
image3446×174 115 KB

看起来是内存资源不足，你白屏安装的时候也没有手动设置内存之类的参数吧？
obd cluster list;
obd cluster edit-config 部署名;看看配置

#12

刚刚我把内存从81G改到88G， 然后reload 还是没有能启动， 我再次上传LOG了。
配置如下：

[oceanbase@localhost ~]$ obd cluster list
±-----------------------------------------------------------+
| Cluster List |
±-----±----------------------------------±----------------+
| Name | Configuration Path | Status (Cached) |
±-----±----------------------------------±----------------+
| fund | /home/oceanbase/.obd/cluster/fund | stopped |
±-----±----------------------------------±----------------+
Trace ID: 0de982a8-d9b0-11ed-9b4b-00155d1f7e00
If you want to view detailed obd logs, please run: obd display-trace 0de982a8-d9b0-11ed-9b4b-00155d1f7e00

observer.zip (3.0 MB)

[oceanbase@localhost ~]$ obd cluster edit-config fund
user:
username: oceanbase
password: aja6@123
port: 22
oceanbase-ce:
version: 4.1.0.0
release: 100000192023032010.el8
package_hash: 1fcbef263fe849d1876d9d6f65c45208d0b29844
10.0.10.207:
zone: zone1
servers:

  • 10.0.10.207
    global:
    appname: fund
    root_password: AbcD123!@#
    mysql_port: 2881
    rpc_port: 2882
    home_path: /home/oceanbase/fund/oceanbase
    ocp_meta_tenant_log_disk_size: 6G
    enable_syslog_recycle: true
    enable_syslog_wf: false
    max_syslog_file_count: 4
    memory_limit: 88G
    datafile_size: 800G
    datafile_disk_percentage: 90
    data_disk_usage_limit_percentage: 95
    system_memory: 18G
    log_disk_size: 500G
    log_disk_percentage: 90
    log_disk_utilization_threshold: 95
    cpu_count: 46
    obproxy-ce:
    version: 4.1.0.0
    package_hash: b7751f64893bb2f66a018afeeb7276865fc88f30
    release: 7.el8
    servers:
  • 10.0.10.207
    global:
    prometheus_listen_port: 2884
    listen_port: 2883
    home_path: /home/oceanbase/fund/obproxy
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    enable_cluster_checkout: false
    depends:
  • oceanbase-ce
    obagent:
    version: 1.3.0
    package_hash: 647924a34ac989ce17b9e2d58e942ce3268b3b8e
    release: 22.el8
    servers:
  • 10.0.10.207
    global:
    monagent_http_port: 8088
    mgragent_http_port: 8089
    home_path: /home/oceanbase/fund/obagent
    ob_monitor_status: active
    depends:
  • oceanbase-ce
    ocp-express:
    version: 1.0.0
    package_hash: 093d48d4339920b567f02a7ada83891c86deaf65
    release: 100000432023032015.el8
    servers:
  • 10.0.10.207
    global:
    port: 8180
    home_path: /home/oceanbase/fund/ocpexpress
    memory_size: 752M
    depends:
  • obagent
  • oceanbase-ce
  • obproxy-ce
#13

你试下重新部署，刚刚初始化已经失败了，改了配置文件以后，不会再初始化的；
obd cluster redeploy fund
试试

#14

感谢感谢！！！，redeploy 这样原来数据会丢失了？

#15

redeploy集群是删除再重建，数据会丢失的；你这是新建的吧，可以redeploy吧？
你这个主要是因为集群初始化失败了，即使改了参数，reload也不会再触发初始化，所以需要重新部署一下；后续使用的时候如果只是改参数，reload生效就行，不需要redeploy

#16

谢谢！！！ 已经搞定了， 是因为硬盘空间不够。
把日志大小改大后， 再reload 就好了。

#17

如果把日志和数据分别移到到两个硬盘里， 在现在数据不丢的情况下， 怎么来处理了？ 谢谢！

#18

data_dir和redo_dir重新设置的话需要redeploy的，可以参考一下这个文件的参数说明，有说明修改生效的条件：https://github.com/oceanbase/obdeploy/blob/master/plugins/oceanbase/4.0.0.0/parameter.yaml

image
image862×728 76.5 KB

#19

好的， 谢谢！！ 如果用目录软连接还可以了？

#20

建议还是按实际规划建目录分开吧