docker启动，oceanbase-ce挂载卷后，删除容器，重启容器报错

[egegredsfew]

【 使用环境 】 测试环境
【 OB or 其他组件 】ob obd
【 使用版本 】 oceanbase/oceanbase-ce:latest
【问题描述】docker启动，oceanbase-ce挂载卷后，删除容器，重启容器报错
【复现路径】docker启动，oceanbase-ce挂载卷后，删除容器，重启容器报错
【问题现象及影响】

我使用 docker启动 的
docker run -p 2881:2881 --name obstandalone -v /home/zjhe/oceanbase/ob:/root/ob -v /home/zjhe/oceanbase/obd:/root/.obd -e MODE=mini -e OB_MEMORY_LIMIT=2G -e OB_SYSTEM_MEMORY=500m -e OB_ROOT_PASSWORD=wODvkWcPxBVTQU7w -d oceanbase/oceanbase-ce:latest
然后停止容器
删除容器，
再次重新创建容器启动，就报错。
遇到两种报错 ：

1、 [DEBUG] – local execute: cd /root/ob; /root/ob/bin/observer -r ‘127.0.0.1:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -n ‘obcluster’ -c 1 -d ‘/root/ob/store’ -i ‘lo’ -l ‘INFO’ -o __min_full_resource_pool_memory=2147483648,memory_limit=‘3G’,system_memory=‘1G’,datafile_size=‘5G’,log_disk_size=‘5G’,cpu_count=16,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4
[2023-11-01 02:47:21.058] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] – exited code 0
[2023-11-01 02:47:21.059] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] – root@127.0.0.1 delete env LD_LIBRARY_PATH
[2023-11-01 02:47:21.059] [fa22f74a-7860-11ee-b1a6-0242ac110002] [INFO] observer program health check
[2023-11-01 02:47:24.063] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] – 127.0.0.1 program health check
[2023-11-01 02:47:24.063] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] – local execute: cat /root/ob/run/observer.pid
[2023-11-01 02:47:24.066] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] – exited code 1, error output:
[2023-11-01 02:47:24.066] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] cat: /root/ob/run/observer.pid: No such file or directory
[2023-11-01 02:47:24.066] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG]
[2023-11-01 02:47:24.067] [fa22f74a-7860-11ee-b1a6-0242ac110002] [WARNING] OBD-2002: Failed to start 127.0.0.1 observer
[2023-11-01 02:47:24.067] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] - sub start ref count to 0
[2023-11-01 02:47:24.067] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] - export start
[2023-11-01 02:47:24.067] [fa22f74a-7860-11ee-b1a6-0242ac110002] [ERROR] oceanbase-ce start failed
[2023-11-01 02:47:24.069] [fa22f74a-7860-11ee-b1a6-0242ac110002] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-11-01 02:47:24.069] [fa22f74a-7860-11ee-b1a6-0242ac110002] [INFO] Trace ID: fa22f74a-7860-11ee-b1a6-0242ac110002
[2023-11-01 02:47:24.069] [fa22f74a-7860-11ee-b1a6-0242ac110002] [INFO] If you want to view detailed obd logs, please run: obd display-trace fa22f74a-7860-11ee-b1a6-0242ac110002
[2023-11-01 02:47:24.069] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2023-11-01 02:47:24.069] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2023-11-01 02:47:24.070] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2023-11-01 02:47:24.070] [fa22f74a-7860-11ee-b1a6-0242ac110002] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obcluster release, count 0

2、[8b323b9a-785e-11ee-a8bb-6add6a2eabef] [ERROR] _errno.LockError: [Errno 9] Bad file descriptor
[2023-11-01 02:29:55.282] [8b323b9a-785e-11ee-a8bb-6add6a2eabef] [ERROR]
[2023-11-01 02:29:55.282] [8b323b9a-785e-11ee-a8bb-6add6a2eabef] [INFO] Trace ID: 8b323b9a-785e-11ee-a8bb-6add6a2eabef
[2023-11-01 02:29:55.282] [8b323b9a-785e-11ee-a8bb-6add6a2eabef] [INFO] If you want to view detailed obd logs, please run: obd display-trace 8b323b9a-785e-11ee-a8bb-6add6a2eabef
[2023-11-01 02:29:55.282] [8b323b9a-785e-11ee-a8bb-6add6a2eabef] [DEBUG] - unlock /root/.obd/lock/global
[2023-11-01 02:29:56.310] [8bd19de8-785e-11ee-a9a4-6add6a2eabef] [DEBUG] - cmd: []
[2023-11-01 02:29:56.311] [8bd19de8-785e-11ee-a9a4-6add6a2eabef] [DEBUG] - opts: {}
[2023-11-01 02:29:56.311] [8bd19de8-785e-11ee-a9a4-6add6a2eabef] [DEBUG] - mkdir /root/.obd/lock/
…skipping…
[2023-11-01 02:52:48.369] [bda0a000-7861-11ee-9d9c-4e37ab7c9d6c] [ERROR] _errno.LockError: [Errno 9] Bad file descriptor
[2023-11-01 02:52:48.369] [bda0a000-7861-11ee-9d9c-4e37ab7c9d6c] [ERROR]
[2023-11-01 02:52:48.369] [bda0a000-7861-11ee-9d9c-4e37ab7c9d6c] [INFO] Trace ID: bda0a000-7861-11ee-9d9c-4e37ab7c9d6c
[2023-11-01 02:52:48.369] [bda0a000-7861-11ee-9d9c-4e37ab7c9d6c] [INFO] If you want to view detailed obd logs, please run: obd display-trace bda0a000-7861-11ee-9d9c-4e37ab
7c9d6c
[2023-11-01 02:52:48.369] [bda0a000-7861-11ee-9d9c-4e37ab7c9d6c] [DEBUG] - unlock /root/.obd/lock/global
[2023-11-01 02:52:49.335] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [DEBUG] - cmd: [‘obcluster’]
[2023-11-01 02:52:49.335] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [DEBUG] - opts: {‘tenant_name’: ‘test’, ‘max_cpu’: None, ‘min_cpu’: None, ‘max_memory’: None, ‘min_memory’: None, ‘memory_size’: None, ‘max_disk_size’: None, ‘log_disk_size’: None, ‘max_iops’: None, ‘min_iops’: None, ‘iops_weight’: None, ‘max_session_num’: None, ‘unit_num’: None, ‘zone_list’: None, ‘mode’: ‘mysql’, ‘charset’: None, ‘collate’: None, ‘replica_num’: None, ‘logonly_replica_num’: None, ‘tablegroup’: None, ‘primary_zone’: ‘RANDOM’, ‘locality’: None, ‘variables’: “ob_tcp_invited_nodes=’%’”}
[2023-11-01 02:52:49.335] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [DEBUG] - mkdir /root/.obd/lock/
[2023-11-01 02:52:49.336] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [DEBUG] - unknown lock mode
[2023-11-01 02:52:49.337] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] Another app is currently holding the obd lock.
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] Traceback (most recent call last):
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “_lock.py”, line 71, in _sh_lock
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “tool.py”, line 483, in share_lock_obj
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] OSError: [Errno 9] Bad file descriptor
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR]
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] During handling of the above exception, another exception occurred:
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR]
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] Traceback (most recent call last):
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “obd.py”, line 234, in do_command
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “core.py”, line 75, in init
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “_lock.py”, line 268, in global_sh_lock
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “_lock.py”, line 259, in _sh_lock
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “_lock.py”, line 254, in _lock
[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “_lock.py”, line 176, in lock
[2023-11-01 02:52:49.339] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “_lock.py”, line 77, in sh_lock
[2023-11-01 02:52:49.339] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] File “_lock.py”, line 73, in _sh_lock
[2023-11-01 02:52:49.339] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] _errno.LockError: [Errno 9] Bad file descriptor

【附件】

[甫瑜]

挂载的数据目录都清空下试试

[linguanren]

清空你的挂载目录，数据盘还在，deploy会失败

[egegredsfew]

挂载目录清空了，我的数据不就没了， oceanbase-ce mini 单节点，这种不能做数据持久化么？

[王利博]

1、 memory_limit=‘3G’ 这个有点低 调高下
快速体验 OceanBase 数据库-内存使用建议
2.[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] Another app is currently holding the obd lock. 可以看下是否有进程在占用着

[linguanren]

能，但是docker run指令触发的是deploy的逻辑，而不是restart逻辑
正常stop容器就好，或者可以魔改一下

[egegredsfew]

2.[2023-11-01 02:52:49.338] [be3474a6-7861-11ee-9e7e-4e37ab7c9d6c] [ERROR] Another app is currently holding the obd lock. 可以看下是否有进程在占用着

这个就是 最奇怪的。我首次启动容器，就报这个错误、 我进入容器内部 执行 obd的指令也报错。 ps -ef|grep obd 没有看到有任务

[2023-11-01 06:04:03.528] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [DEBUG] - cmd: [’/root/pkg/obagent-4.2.0-100000062023080210.el7.x86_64.rpm’, ‘/root/pkg/oceanbase-ce-4.2.0
.0-100010032023083021.el7.x86_64.rpm’, ‘/root/pkg/oceanbase-ce-libs-4.2.0.0-100010032023083021.el7.x86_64.rpm’]
[2023-11-01 06:04:03.529] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [DEBUG] - opts: {‘force’: None}
[2023-11-01 06:04:03.529] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [DEBUG] - mkdir /root/.obd/lock/
[2023-11-01 06:04:03.532] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [DEBUG] - unknown lock mode
[2023-11-01 06:04:03.536] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] Another app is currently holding the obd lock.
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] Traceback (most recent call last):
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “_lock.py”, line 71, in _sh_lock
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “tool.py”, line 483, in share_lock_obj
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] OSError: [Errno 9] Bad file descriptor
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR]
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] During handling of the above exception, another exception occurred:
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR]
[2023-11-01 06:04:03.538] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] Traceback (most recent call last):
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “obd.py”, line 234, in do_command
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “core.py”, line 75, in init
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “_lock.py”, line 268, in global_sh_lock
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “_lock.py”, line 259, in _sh_lock
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “_lock.py”, line 254, in _lock
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “_lock.py”, line 176, in lock
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “_lock.py”, line 77, in sh_lock
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] File “_lock.py”, line 73, in _sh_lock
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR] _errno.LockError: [Errno 9] Bad file descriptor
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [ERROR]
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [INFO] Trace ID: 755a5a78-787c-11ee-ba50-7a63d71f7f88
[2023-11-01 06:04:03.539] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [INFO] If you want to view detailed obd logs, please run: obd display-trace 755a5a78-787c-11ee-ba50-7a63d7
1f7f88
[2023-11-01 06:04:03.540] [755a5a78-787c-11ee-ba50-7a63d71f7f88] [DEBUG] - unlock /root/.obd/lock/global
[2023-11-01 06:04:04.531] [75f48062-787c-11ee-827c-7a63d71f7f88] [DEBUG] - cmd: []
[2023-11-01 06:04:04.531] [75f48062-787c-11ee-827c-7a63d71f7f88] [DEBUG] - opts: {}
[2023-11-01 06:04:04.531] [75f48062-787c-11ee-827c-7a63d71f7f88] [DEBUG] - mkdir /root/.obd/lock/
[2023-11-01 06:04:04.532] [75f48062-787c-11ee-827c-7a63d71f7f88] [DEBUG] - set lock mode to NO_LOCK(0)
[2023-11-01 06:04:04.532] [75f48062-787c-11ee-827c-7a63d71f7f88] [DEBUG] - Get deploy list
[2023-11-01 06:04:04.532] [75f48062-787c-11ee-827c-7a63d71f7f88] [DEBUG] - mkdir /root/.obd/cluster/
[2023-11-01 06:04:04.560] [75f48062-787c-11ee-827c-7a63d71f7f88] [DEBUG] - mkdir /root/.obd/config_parser/
[2023-11-01 06:04:04.563] [75f48062-787c-11ee-827c-7a63d71f7f88] [INFO] Local deploy is empty
[2023-11-01 06:04:04.563] [75f48062-787c-11ee-827c-7a63d71f7f88] [INFO] Trace ID: 75f48062-787c-11ee-827c-7a63d71f7f88
[2023-11-01 06:04:04.563] [75f48062-787c-11ee-827c-7a63d71f7f88] [INFO] If you want to view detailed obd logs, please run: obd display-trace 75f48062-787c-11ee-827c-7a63d7
1f7f88
[2023-11-01 06:04:05.873] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [DEBUG] - cmd: []
[2023-11-01 06:04:05.873] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [DEBUG] - opts: {}
[2023-11-01 06:04:05.873] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [DEBUG] - mkdir /root/.obd/lock/
[2023-11-01 06:04:05.873] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [DEBUG] - unknown lock mode
[2023-11-01 06:04:05.874] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] Another app is currently holding the obd lock.
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] Traceback (most recent call last):
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “_lock.py”, line 71, in _sh_lock
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “tool.py”, line 483, in share_lock_obj
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] OSError: [Errno 9] Bad file descriptor
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR]
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] During handling of the above exception, another exception occurred:
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR]
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] Traceback (most recent call last):
[2023-11-01 06:04:05.875] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “obd.py”, line 234, in do_command
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “core.py”, line 75, in init
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “_lock.py”, line 268, in global_sh_lock
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “_lock.py”, line 259, in _sh_lock
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “_lock.py”, line 254, in _lock
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “_lock.py”, line 176, in lock
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “_lock.py”, line 77, in sh_lock
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] File “_lock.py”, line 73, in _sh_lock
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR] _errno.LockError: [Errno 9] Bad file descriptor
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [ERROR]
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [INFO] Trace ID: 76c1376a-787c-11ee-9a1e-7a63d71f7f88
[2023-11-01 06:04:05.876] [76c1376a-787c-11ee-9a1e-7a63d71f7f88] [INFO] If you want to view detailed obd logs, please run: obd display-trace 76c1376a-787c-11ee-9a1e-7a63d7
1f7f88

[linguanren]

@egegredsfew

.obd

将挂载.obd目录下面的lock文件夹里面的删除

[linguanren]

可以直接删除挂载目录重新起一个新的observer
因为最开始写的时候没有考虑过这样的场景，想成功需要修改源代码