docker无法重启oceanbase/oceanbase-ce:4.1.0.0

OceanBase 技术问题
,
#1

【 使用环境 】生产环境 or 测试环境
系统：centos7
docker version：24.0.4
oceanbase version：4.1.0.0
【 OB or 其他组件 】
【 使用版本 】
【问题描述】清晰明确描述问题
重启命令：
docker run -d -p 2881:2881 -v /oss_oceanbase/root/ob:/root/ob -v /oss_oceanbase/root/obd:/root/.obd --name oss_oceanbase --rm oceanbase/oceanbase-ce:4.1.0.0

错误日志：
find obd deploy information, skip configuring…
start ob cluster …
Get local repositories ok
Search plugins ok
Open ssh connection ok
Load cluster param plugin ok
Cluster status check ok
Check before start observer ok
[WARN] OBD-1011: (127.0.0.1) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)

Check before start obagent ok
Start observer ok
observer program health check ok
Connect to observer ok
Start obagent ok
[ERROR] failed to set config to 127.0.0.1 obagent.
[ERROR] obagent start failed
±--------------------------------------------+
| observer |
±----------±--------±-----±------±-------+
| ip | version | port | zone | status |
±----------±--------±-----±------±-------+
| 127.0.0.1 | 4.1.0.0 | 2881 | zone1 | ACTIVE |
±----------±--------±-----±------±-------+
obclient -h127.0.0.1 -P2881 -uroot -Doceanbase -A

See https://www.oceanbase.com/product/ob-deployer/error-codes .
Trace ID: ba58d78a-30dd-11ee-8d3c-0242ac110002
If you want to view detailed obd logs, please run: obd display-trace ba58d78a-30dd-11ee-8d3c-0242ac110002
boot failed!

【复现路径】问题出现前后相关操作
【问题现象及影响】

【附件】

#2

obd日志
obd.txt (267.0 KB)

#3

observer.log相关错误信息：
[2023-08-02 02:32:53.190521] ERROR issue_dba_error (ob_log.cpp:1786) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=0, file=“ob_tenant_mgr.cpp”, line_no=441, info="
the hold of observer tenant is over the system_memory")
[2023-08-02 02:32:53.190528] EDIAG [COMMON] print_tenant_usage_ (ob_tenant_mgr.cpp:441) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] the hold of observer tenant is over the system_memory(observer_tenant_hold=1201127424, system_memory=1073741824)
BACKTRACE:0xd5f30be 0x59b549e 0x59b5362 0x5997072 0xd14b512 0xd14afae 0xacae760 0xacae0d3 0x3bf8fa5 0xdc671e7 0xdc6402a 0x7f2bc7330ea5 0x7f2bc705996d
[2023-08-02 02:32:53.190609] INFO [LIB] operator() (ob_malloc_allocator.cpp:498) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant: 999, limit: 2,147,483,648 hold: 10,485,760 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0
[MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807
[MEMORY] ctx_id= CO_STACK hold_bytes= 8,388,608 limit= 9,223,372,036,854,775,807
[2023-08-02 02:32:53.190616] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:169) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21]
[MEMORY] tenant_id= 999 ctx_id= DEFAULT_CTX_ID hold= 2,097,152 used= 50,592 limit= 9,223,372,036,854,775,807
[MEMORY] idle_size= 0 free_size= 0
[MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0
[MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=thread_factor
[MEMORY] hold= 6,496 used= 5,600 count= 14 avg_used= 400 block_cnt= 2 chunk_cnt= 1 mod=OMT_Worker
[MEMORY] hold= 3,136 used= 1,232 count= 28 avg_used= 44 block_cnt= 2 chunk_cnt= 1 mod=Coro
[MEMORY] hold= 50,592 used= 43,864 count= 43 avg_used= 1,020 mod=SUMMARY
[2023-08-02 02:32:53.190650] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:169) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21]
[MEMORY] tenant_id= 999 ctx_id= CO_STACK hold= 8,388,608 used= 7,110,656 limit= 9,223,372,036,854,775,807
[MEMORY] idle_size= 0 free_size= 0
[MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0
[MEMORY] hold= 7,110,656 used= 7,097,104 count= 14 avg_used= 506,936 block_cnt= 14 chunk_cnt= 4 mod=CoStack
[MEMORY] hold= 7,110,656 used= 7,097,104 count= 14 avg_used= 506,936 mod=SUMMARY
[2023-08-02 02:32:53.190668] INFO [LIB] operator() (ob_malloc_allocator.cpp:498) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant: 506, limit: 4,294,967,296 hold: 16,777,216 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0
[MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807
[MEMORY] ctx_id= CO_STACK hold_bytes= 14,680,064 limit= 9,223,372,036,854,775,807
[2023-08-02 02:32:53.190674] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:169) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21]
[MEMORY] tenant_id= 506 ctx_id= DEFAULT_CTX_ID hold= 2,097,152 used= 58,864 limit= 9,223,372,036,854,775,807
[MEMORY] idle_size= 0 free_size= 0
[MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0
[MEMORY] hold= 40,960 used= 37,032 count= 1 avg_used= 37,032 block_cnt= 1 chunk_cnt= 1 mod=thread_factor
[MEMORY] hold= 12,080 used= 10,400 count= 26 avg_used= 400 block_cnt= 3 chunk_cnt= 1 mod=OMT_Worker
[MEMORY] hold= 5,824 used= 2,288 count= 52 avg_used= 44 block_cnt= 3 chunk_cnt= 1 mod=Coro
[MEMORY] hold= 58,864 used= 49,720 count= 79 avg_used= 629 mod=SUMMARY
[2023-08-02 02:32:53.190709] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:169) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21]
[MEMORY] tenant_id= 506 ctx_id= CO_STACK hold= 14,680,064 used= 13,205,504 limit= 9,223,372,036,854,775,807
[MEMORY] idle_size= 0 free_size= 0
[MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0
[MEMORY] hold= 13,205,504 used= 13,180,336 count= 26 avg_used= 506,936 block_cnt= 26 chunk_cnt= 7 mod=CoStack
[MEMORY] hold= 13,205,504 used= 13,180,336 count= 26 avg_used= 506,936 mod=SUMMARY
[2023-08-02 02:32:53.190726] INFO [LIB] operator() (ob_malloc_allocator.cpp:498) [235][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=21] [MEMORY] tenant: 508, limit: 1,073,741,824 hold: 18,874,368 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0
[MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 2,097,152 limit= 9,223,372,036,854,775,807
[MEMORY] ctx_id= CO_STACK hold_bytes= 16,777,216 limit= 9,223,372,036,854,775,807

#4

发一下OBD的配置文件

#5

config.zip (1.7 KB)

#6

system_memory 设置为2G试一下呢

#8

看obd配置文件，obagent的端口号在1.3.0版本后从server_port、pprof_port改成了mgragent_http_port、monagent_http_port。是配置文件的问题