【 使用环境 】生产环境
【 OB or 其他组件 】
ob 4.2.1
oms
版本号: 4.2.3_CE
【 使用版本 】
【问题描述】
频繁非预期退出且手工启动失败。导致同步数据问题一直阻塞
10.88.24.178-7100:OB_MYSQL_CE_np_5sphlh4zt3b4_5sphlwo5kwgw-1-0:0000000007 最早拉取位点：2024-06-07 23:59:57
当前拉取位点：2024-06-13 00:00:12
延迟时间：10 小时 非预期退出 启动
更新
查看日志
【复现路径】问题出现前后相关操作
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接（右键跳转查看）：
需要哪个组件？这个论坛只能上传10M，不能所有都传上来
[2024-06-13 07:02:13.349345] INFO [TLOG.FETCHER] run (ob_log_fetcher_idle_pool.cpp:188) [57586][][T0][Y7620A5818B2-0000000000000001-0-0] [lt=19] fetcher idle pool thread exits(thread_index=0, ret=-4114, ret=“OB_IN_STOP_STATE”)
这个时间点是否有人主动停止？
如果不是，请查看一下系统资源情况，是否是由于容器资源不够被 操作系统强杀了
应该是内存满了进程自动被停了，现在怎么样让这个任务再起起来？手工点了启动没有作用
先在容器中根据store端口查找一下进程，如果在的话，kill -9 一下，再手动点启动
或者新建一个：
打开store的配置，找到subtopic这个配置项，复制一下
再去“运维监控”-“组件” 创建一个store，位点选择之前那个store下游增量位点时间 前10分钟即可
增量任务显示位点时间是:
当前位点：2024-06-11 00:11:45
新增加的store，显示
起始位点：2024-06-08 22:59:49
当前位点：2024-06-08 23:01:37
连接数：0
这个新增加store操作之前操作过，这个位点离增量的时间差了3天，等这个store追数据到离最近时间相近的时候，中间会因为内存问题退出。再重建store还是从这个位点开始，这样问题解决不了
新建store的时候，时间是可以选的，根据增量任务时间点 设置
创建的时候是按这个时间创建的
2024-06-11 00:00:00
实际上创建出来的时候的位点起始时间，不是这个时间，
实际看到的时间是：
当前位点：2024-06-08 23:01:37
而且不管创建的时候用哪个时间，创建出来 新的store一直是这个时间
前面停掉的store截图看一下，它现在的位点是什么？
有没有试过第一种方法，在oms容器中找一下之前停掉store进程是否还在，在的话kill -9，然后再点击启动
之前意外退出的store,显示是,2024-06-13 00:00:12
|10.88.24.178-7100:OB_MYSQL_CE_np_5sphlh4zt3b4_5sphlwo5kwgw-1-0:0000000007|OceanBase-CE|0|52,761|非预期退出|启动
更新
查看日志
| 更多 |
|---|
| 起始位点：2024-06-07 23:59:57 |
当前位点：2024-06-13 00:00:12
连接数：0|
kill -9 store试过，再点重启没有用，起不来
链路-查看组件监控 截图看一下
oms容器中 ps -ef|grep 7100
截图看一下
ps -ef|grep 7100
ds 5942 1 0 14:44 ? 00:00:10 ./bin/store -c /home/ds/store/store7100/conf/stores.conf
root 34700 34372 0 15:29 ? 00:00:00 grep --color=auto 7100
这不是还在后台吗，kill -9 5942
确认进程不在了，再在页面上点击启动
这是下午14:44 进行了kill -9 以后新启动的进程，任务根本没起来，这样操作没有用。
这样操作之后，最新的libobcdc.log传一下
libobcdc.log
[2024-06-13 07:02:37.104888] INFO eloop_run (eloop.c:127) [57538][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1718233357104885, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0
[2024-06-13 07:02:37.111878] INFO eloop_run (eloop.c:127) [57542][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1718233357111876, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0
[2024-06-13 07:02:38.112501] INFO eloop_run (eloop.c:127) [57538][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1718233358112500, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0
[2024-06-13 07:02:38.119498] INFO eloop_run (eloop.c:127) [57542][pnio2][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1718233358119494, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0
[2024-06-13 07:02:38.370855] INFO unregister_pm (ob_page_manager.cpp:50) [57259][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] unregister pm finish(&pm=0x7f68543f5d80, pm.get_tid()=57259)
[2024-06-13 07:02:38.371825] INFO unregister_pm (ob_page_manager.cpp:50) [56891][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=59] unregister pm finish(&pm=0x7f685427bd80, pm.get_tid()=56891)
[2024-06-13 07:02:38.371924] INFO [TLOG] do_destroy_ (ob_log_instance.cpp:1435) [56889][][T0][Y0-0000000000000000-0-0] [lt=13] destroy obcdc end
[2024-06-13 07:02:38.371954] INFO [TLOG] destroy_instance (ob_log_instance.cpp:127) [56889][][T0][Y0-0000000000000000-0-0] [lt=12] ObLogInstance 0x7f6896e17010 destroy
[2024-06-13 07:02:38.371960] INFO [TLOG] ~ObLogInstance (ob_log_instance.cpp:214) [56889][][T0][Y0-0000000000000000-0-0] [lt=5] ====================libobcdc end====================
[2024-06-13 07:02:38.373415] INFO destroy_array (ob_dchash.h:424) [56889][][T0][Y0-0000000000000000-0-0] [lt=8] DCHash: destroy_array: N9oceanbase6common9sqlclient12TenantMapKeyE this=0x7f6896e39e90 array=0x7f6824804030 array_size=65536
[root@infra-bjmt-rs-97 log]# tail -100 libobcdc.log
[2024-06-13 07:02:35.315775] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf88f30, slice=0x7f66dcf88f40)
[2024-06-13 07:02:35.315778] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4016] (item=0x7f66dcf88f70, slice=0x7f66dcf88f80)
[2024-06-13 07:02:35.315781] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf88fb0, slice=0x7f66dcf88fc0)
[2024-06-13 07:02:35.315785] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf88ff0, slice=0x7f66dcf89000)
[2024-06-13 07:02:35.315788] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4016] (item=0x7f66dcf89030, slice=0x7f66dcf89040)
[2024-06-13 07:02:35.315791] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89070, slice=0x7f66dcf89080)
[2024-06-13 07:02:35.315795] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf890b0, slice=0x7f66dcf890c0)
[2024-06-13 07:02:35.315798] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf890f0, slice=0x7f66dcf89100)
[2024-06-13 07:02:35.315801] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89130, slice=0x7f66dcf89140)
[2024-06-13 07:02:35.315805] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89170, slice=0x7f66dcf89180)
[2024-06-13 07:02:35.315808] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf891b0, slice=0x7f66dcf891c0)
[2024-06-13 07:02:35.315813] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4016] (item=0x7f66dcf891f0, slice=0x7f66dcf89200)
[2024-06-13 07:02:35.315817] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89230, slice=0x7f66dcf89240)
[2024-06-13 07:02:35.315820] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89270, slice=0x7f66dcf89280)
[2024-06-13 07:02:35.315824] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf892b0, slice=0x7f66dcf892c0)
[2024-06-13 07:02:35.315827] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf892f0, slice=0x7f66dcf89300)
[2024-06-13 07:02:35.315831] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89330, slice=0x7f66dcf89340)
[2024-06-13 07:02:35.315834] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89370, slice=0x7f66dcf89380)
[2024-06-13 07:02:35.315838] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf893b0, slice=0x7f66dcf893c0)
[2024-06-13 07:02:35.315841] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf893f0, slice=0x7f66dcf89400)
[2024-06-13 07:02:35.315845] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89430, slice=0x7f66dcf89440)
[2024-06-13 07:02:35.315848] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89470, slice=0x7f66dcf89480)
[2024-06-13 07:02:35.315852] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf894b0, slice=0x7f66dcf894c0)
[2024-06-13 07:02:35.315855] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf894f0, slice=0x7f66dcf89500)
[2024-06-13 07:02:35.315859] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89530, slice=0x7f66dcf89540)
[2024-06-13 07:02:35.315862] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89570, slice=0x7f66dcf89580)
[2024-06-13 07:02:35.315866] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf895b0, slice=0x7f66dcf895c0)
[2024-06-13 07:02:35.315870] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf895f0, slice=0x7f66dcf89600)
[2024-06-13 07:02:35.315873] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89630, slice=0x7f66dcf89640)
[2024-06-13 07:02:35.315877] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89670, slice=0x7f66dcf89680)
[2024-06-13 07:02:35.315880] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf896b0, slice=0x7f66dcf896c0)
[2024-06-13 07:02:35.315884] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf896f0, slice=0x7f66dcf89700)
[2024-06-13 07:02:35.315888] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89730, slice=0x7f66dcf89740)
[2024-06-13 07:02:35.315893] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=4][errcode=-4016] (item=0x7f66dcf89770, slice=0x7f66dcf89780)
[2024-06-13 07:02:35.315897] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf897b0, slice=0x7f66dcf897c0)
[2024-06-13 07:02:35.315900] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4016] (item=0x7f66dcf897f0, slice=0x7f66dcf89800)
[2024-06-13 07:02:35.315904] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89830, slice=0x7f66dcf89840)
[2024-06-13 07:02:35.315907] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89870, slice=0x7f66dcf89880)
[2024-06-13 07:02:35.315911] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf898b0, slice=0x7f66dcf898c0)
[2024-06-13 07:02:35.315915] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf898f0, slice=0x7f66dcf89900)
[2024-06-13 07:02:35.315918] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89930, slice=0x7f66dcf89940)
[2024-06-13 07:02:35.315922] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89970, slice=0x7f66dcf89980)
[2024-06-13 07:02:35.315926] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf899b0, slice=0x7f66dcf899c0)
[2024-06-13 07:02:35.315930] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf899f0, slice=0x7f66dcf89a00)
[2024-06-13 07:02:35.315933] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4016] (item=0x7f66dcf89a30, slice=0x7f66dcf89a40)
[2024-06-13 07:02:35.315937] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89a70, slice=0x7f66dcf89a80)
[2024-06-13 07:02:35.315941] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89ab0, slice=0x7f66dcf89ac0)
[2024-06-13 07:02:35.315944] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89af0, slice=0x7f66dcf89b00)
[2024-06-13 07:02:35.315948] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89b30, slice=0x7f66dcf89b40)
[2024-06-13 07:02:35.315952] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89b70, slice=0x7f66dcf89b80)
[2024-06-13 07:02:35.315956] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89bb0, slice=0x7f66dcf89bc0)
[2024-06-13 07:02:35.315959] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89bf0, slice=0x7f66dcf89c00)
[2024-06-13 07:02:35.315963] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89c30, slice=0x7f66dcf89c40)
[2024-06-13 07:02:35.315967] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89c70, slice=0x7f66dcf89c80)
[2024-06-13 07:02:35.315971] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89cb0, slice=0x7f66dcf89cc0)
[2024-06-13 07:02:35.315976] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4016] (item=0x7f66dcf89cf0, slice=0x7f66dcf89d00)
[2024-06-13 07:02:35.315980] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89d30, slice=0x7f66dcf89d40)
[2024-06-13 07:02:35.315984] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89d70, slice=0x7f66dcf89d80)
[2024-06-13 07:02:35.315988] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89db0, slice=0x7f66dcf89dc0)
[2024-06-13 07:02:35.315992] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89df0, slice=0x7f66dcf89e00)
[2024-06-13 07:02:35.315996] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89e30, slice=0x7f66dcf89e40)
[2024-06-13 07:02:35.316000] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] (item=0x7f66dcf89e70, slice=0x7f66dcf89e80)
[2024-06-13 07:02:35.316003] WDIAG [LIB] print_leak_slice (ob_slice_alloc.cpp:40) [56889][][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4016] (item=0x7f66dcf89eb0, slice=0x7f66dcf89ec0)
[2024-06-13 07:02:35.316007] EDIAG [LIB] destroy (ob_slice_alloc.cpp:72) [56889][][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4016] there was memory leak, stock=1073741911, total=104, remain=88 BACKTRACE:0x16baee5d 0xb2d13b6 0xb2d0e22 0xb2d0a3c 0x17eb761b 0x17eb7523 0xb309470 0xb70c605 0xb683b48 0xb683e7e 0xb3bcfe5 0xb3b6ccf 0x7f689c22231a 0x7f68a6cd0b62 0x7f68a6655ea5 0x7f68a5c6fb0d
[2024-06-13 07:02:35.316512] INFO [TLOG] stop (ob_log_storager.cpp:125) [56889][][T0][Y0-0000000000000000-0-0] [lt=21] stop storager threads succ(thread_num=10)
[2024-06-13 07:02:35.316522] INFO [TLOG] destroy (ob_log_storager.cpp:88) [56889][][T0][Y0-0000000000000000-0-0] [lt=10] store_service destroy begin
[2024-06-13 07:02:35.316557] INFO [TLOG] destroy (ob_log_storager.cpp:100) [56889][][T0][Y0-0000000000000000-0-0] [lt=3] store_service destroy end
[2024-06-13 07:02:35.316578] INFO [TLOG] destroy (ob_log_reader.cpp:89) [56889][][T0][Y0-0000000000000000-0-0] [lt=3] ObLogReader destroy begin
[2024-06-13 07:02:35.316612] INFO [TLOG] destroy (ob_log_reader.cpp:101) [56889][][T0][Y0-0000000000000000-0-0] [lt=4] ObLogReader destroy end
[2024-06-13 07:02:35.316620] INFO [TLOG] destroy (ob_log_meta_data_baseline_loader.cpp:60) [56889][][T0][Y0-0000000000000000-0-0] [lt=3] ObLogMetaDataBaselineLoader destroy begin
[2024-06-13 07:02:35.318193] INFO destroy_array (ob_dchash.h:424) [56889][][T0][Y0-0000000000000000-0-0] [lt=3] DCHash: destroy_array: N9oceanbase8libobcdc11MetaDataKeyE this=0x7f683fdc4a00 array=0x7f663c404030 array_size=65536
[2024-06-13 07:02:35.318208] INFO [TLOG] destroy (ob_log_meta_data_baseline_loader.cpp:65) [56889][][T0][Y0-0000000000000000-0-0] [lt=11] ObLogMetaDataBaselineLoader destroy finish
[2024-06-13 07:02:35.318211] INFO [TLOG] destroy_components_ (ob_log_instance.cpp:1366) [56889][][T0][Y0-0000000000000000-0-0] [lt=3] destroy all components end
[2024-06-13 07:02:35.611749] INFO [TLOG.SCHEMA] destroy (ob_log_timezone_info_getter.cpp:148) [56889][][T0][Y0-0000000000000000-0-0] [lt=4] destroy ObCDCTimeZoneInfoGetter begin
[2024-06-13 07:02:35.611790] INFO [TLOG.SCHEMA] destroy (ob_log_timezone_info_getter.cpp:158) [56889][][T0][Y0-0000000000000000-0-0] [lt=32] destroy ObCDCTimeZoneInfoGetter success
[2024-06-13 07:02:35.611797] INFO [COMMON] destroy (ob_kv_storecache.cpp:275) [56889][][T0][Y0-0000000000000000-0-0] [lt=4] Begin destroy the ObKVGlobalCache!
[2024-06-13 07:02:35.611807] INFO stop (ob_timer.cpp:98) [56889][][T0][Y0-0000000000000000-0-0] [lt=3] ObTimer stop success(this=0x7f68541ed7f0, thread_id=56899)
[2024-06-13 07:02:35.611824] INFO cancel_all (ob_timer.cpp:352) [56889][][T0][Y0-0000000000000000-0-0] [lt=12] cancel all(ret=0, this=0x7f68541ed7f0, thread_id=56899, wakeup_time_=8359396765421, tasks_num_=0)
[2024-06-13 07:02:35.611830] INFO stop (ob_timer.cpp:98) [56889][][T0][Y0-0000000000000000-0-0] [lt=6] ObTimer stop success(this=0x7f68541ed960, thread_id=56902)
[2024-06-13 07:02:35.611837] INFO cancel_all (ob_timer.cpp:352) [56889][][T0][Y0-0000000000000000-0-0] [lt=3] cancel all(ret=0, this=0x7f68541ed960, thread_id=56902, wakeup_time_=8359396765422, tasks_num_=0)
[2024-06-13 07:02:35.611862] INFO run1 (ob_timer.cpp:479) [56899][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=11067] timer thread exit(this=0x7f68541ed7f0, thread_id=56899)
[2024-06-13 07:02:35.611900] INFO unregister_pm (ob_page_manager.cpp:50) [56899][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] unregister pm finish(&pm=0x7f68542f9d80, pm.get_tid()=56899)
[2024-06-13 07:02:35.612012] INFO destroy (ob_timer.cpp:139) [56889][][T0][Y0-0000000000000000-0-0] [lt=4] ObTimer destroy(this=0x7f68541ed7f0, thread_id=56899)
[2024-06-13 07:02:35.612031] INFO run1 (ob_timer.cpp:479) [56902][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] timer thread exit(this=0x7f68541ed960, thread_id=56902)
[2024-06-13 07:02:35.612061] INFO unregister_pm (ob_page_manager.cpp:50) [56902][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] unregister pm finish(&pm=0x7f6854377d80, pm.get_tid()=56902)
[2024-06-13 07:02:35.612154] INFO destroy (ob_timer.cpp:139) [56889][][T0][Y0-0000000000000000-0-0] [lt=5] ObTimer destroy(this=0x7f68541ed960, thread_id=56902)
[2024-06-13 07:02:35.764374] INFO [COMMON] destroy (ob_kvcache_hazard_version.cpp:213) [56889][][T0][Y0-0000000000000000-0-0] [lt=5] Hazard version begin to destroy
[2024-06-13 07:02:35.793403] INFO [COMMON] destroy (ob_kv_storecache.cpp:291) [56889][][T0][Y0-0000000000000000-0-0] [lt=14] The ObKVGlobalCache has been destroyed!
[2024-06-13 07:02:36.097274] INFO eloop_run (eloop.c:127) [57538][pnio1][T0][Y0-0000000000000000-0-0] [lt=23] PNIO [ratelimit] time: 1718233356097271, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0
[2024-06-13 07:02:36.104267] INFO eloop_run (eloop.c:127) [57542][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1718233356104264, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0
[2024-06-13 07:02:37.104888] INFO eloop_run (eloop.c:127) [57538][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1718233357104885, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0
[2024-06-13 07:02:37.111878] INFO eloop_run (eloop.c:127) [57542][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1718233357111876, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0
[2024-06-13 07:02:38.112501] INFO eloop_run (eloop.c:127) [57538][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1718233358112500, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0
[2024-06-13 07:02:38.119498] INFO eloop_run (eloop.c:127) [57542][pnio2][T0][Y0-0000000000000000-0-0] [lt=34] PNIO [ratelimit] time: 1718233358119494, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0
[2024-06-13 07:02:38.370855] INFO unregister_pm (ob_page_manager.cpp:50) [57259][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] unregister pm finish(&pm=0x7f68543f5d80, pm.get_tid()=57259)
[2024-06-13 07:02:38.371825] INFO unregister_pm (ob_page_manager.cpp:50) [56891][ClockGenerator][T0][Y0-0000000000000000-0-0] [lt=59] unregister pm finish(&pm=0x7f685427bd80, pm.get_tid()=56891)
[2024-06-13 07:02:38.371924] INFO [TLOG] do_destroy_ (ob_log_instance.cpp:1435) [56889][][T0][Y0-0000000000000000-0-0] [lt=13] destroy obcdc end
[2024-06-13 07:02:38.371954] INFO [TLOG] destroy_instance (ob_log_instance.cpp:127) [56889][][T0][Y0-0000000000000000-0-0] [lt=12] ObLogInstance 0x7f6896e17010 destroy
[2024-06-13 07:02:38.371960] INFO [TLOG] ~ObLogInstance (ob_log_instance.cpp:214) [56889][][T0][Y0-0000000000000000-0-0] [lt=5] ====================libobcdc end====================
[2024-06-13 07:02:38.373415] INFO destroy_array (ob_dchash.h:424) [56889][][T0][Y0-0000000000000000-0-0] [lt=8] DCHash: destroy_array: N9oceanbase6common9sqlclient12TenantMapKeyE this=0x7f6896e39e90 array=0x7f6824804030 array_size=65536
congo_0
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:591 10210,2006660864] load config key:tb_white_list value:merchantMarketing.gifshow.order_quote_bill_mapping|merchantMarketing.gifshow.coupon_receive_buyer|merchantMarketing.gifshow.marketing_budget_refund_flow|merchantMarketing.gifshow.marketing_card_voucher|merchantMarketing.gifshow.mr_acquire|merchantMarketing.gifshow.marketing_budget_use_flow|merchantMarketing.gifshow.cash_assets_trade_refund_record|merchantMarketing.gifshow.order_quote_bill|merchantMarketing.gifshow.promotion_charge|merchantMarketing.gifshow.cash_assets_trade_pay_record|merchantMarketing.gifshow.marketing_card_voucher_flow|merchantMarketing.gifshow.marketing_report_entry|merchantMarketing.gifshow.promotion_activity|merchantMarketing.gifshow.promotion_user
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:591 10210,2006660864] load config key:timezone value:+08:00
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:577 10210,2006660864] Config of sectionnames:partition is empty
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:577 10210,2006660864] Config of sectionnames:hbase2store is empty
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:577 10210,2006660864] Config of sectionnames:oracle2store is empty
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:577 10210,2006660864] Config of sectionnames:deliver2store is empty
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:577 10210,2006660864] Config of sectionnames:unit is empty
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:577 10210,2006660864] Config of sectionnames:db2tostore is empty
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:577 10210,2006660864] Config of sectionnames:logproxy2store is empty
2024-06-13 15:41:00 [NOTICE] [StoreManager.cpp:834 10210,2006660864] succeed to init record sent log with log.sent.record=0, log.sent.record.users=, log.sent.max.files=50
store.log
2024-06-12 13:59:30 (Epoll.cpp:94): Epoll start, use FD: 11
2024-06-13 07:01:31 (ConnectionManager.cpp:77): default buf size is: 1048576, Msg split size is: 131072, Force Use Encrypt: false, Using ThreadPool true
2024-06-13 07:01:31 (ConnectionManager.cpp:139): listen info is: 17001
2024-06-13 07:01:31 (Epoll.cpp:94): Epoll start, use FD: 12
2024-06-13 07:01:31 (Epoll.cpp:94): Epoll start, use FD: 13
2024-06-13 11:23:40 (ConnectionManager.cpp:77): default buf size is: 1048576, Msg split size is: 131072, Force Use Encrypt: false, Using ThreadPool true
2024-06-13 11:23:40 (ConnectionManager.cpp:116): Address already in use, FD: 9
2024-06-13 11:23:53 (ConnectionManager.cpp:187): Connection Manager stop , close FD: 9
2024-06-13 11:23:55 (Epoll.cpp:25): Epoll quit close fd:0
2024-06-13 11:23:57 (Epoll.cpp:25): Epoll quit close fd:0
store.log / libobcdc.log这两个文件就没有更新过，这东西点了以后，就没起起来。
