【 使用环境 】测试环境
vmware 3台observer : 内存9216M , cpu: 2核
【 使用版本 】
社区版v3.1.5
【问题描述】
obd cluster start obtest失败
Get local repositories ok
Search plugins ok
Load cluster param plugin ok
Open ssh connection ok
Check before start observer ok
[WARN] OBD-1007: (192.168.123.160) The recommended number of stack size is unlimited (Current value: 20480)
[WARN] OBD-2000: (192.168.123.160) not enough memory. (Free: 7838M, Need: 8192M)
[WARN] OBD-1007: (192.168.123.161) The recommended number of stack size is unlimited (Current value: 20480)
[WARN] OBD-2000: (192.168.123.161) not enough memory. (Free: 7856M, Need: 8192M)
[WARN] OBD-1007: (192.168.123.162) The recommended number of stack size is unlimited (Current value: 20480)
[WARN] OBD-2000: (192.168.123.162) not enough memory. (Free: 7845M, Need: 8192M)
Start observer ok
observer program health check ok
Connect to observer …
【复现路径】
1、obd安装yaml
global:
devname: ens32
cluster_id: 1
memory_limit: 8G
system_memory: 1G
stack_size: 512K
cpu_count: 2
cache_wash_threshold: 1G
__min_full_resource_pool_memory: 268435456
workers_per_cpu_quota: 6
schema_history_expire_time: 1d
net_thread_count: 4
major_freeze_duty_time: Disable
minor_freeze_times: 10
enable_separate_sys_clog: 0
enable_merge_by_turn: FALSE
datafile_disk_percentage: 20
syslog_level: INFO
enable_syslog_wf: false
enable_syslog_recycle: true
max_syslog_file_count: 4
2、observer.log 日志
[2024-07-16 11:43:16.168951] INFO [COMMON] ob_kvcache_store.cpp:323 [5681][100][Y0-0000000000000000] [lt=3] [dc=0] Wash time detail, (refresh_score_time=90, compute_wash_size_time=71, wash_sort_time=2, wash_time=0)
[2024-07-16 11:43:16.173314] INFO [COMMON] ob_io_disk.cpp:812 [5630][0][Y0-0000000000000000] [lt=14] [dc=0] finish delete disk(ret=0, fd={fd:306, disk_id:{disk_idx:0, install_seq:0}})
[2024-07-16 11:43:16.173374] INFO [STORAGE] ob_store_file.cpp:579 [5630][0][Y0-0000000000000000] [lt=12] [dc=0] The store file will be destroyed!
[2024-07-16 11:43:16.180943] INFO ob_timer.cpp:84 [5630][0][Y0-0000000000000000] [lt=4] [dc=0] ObTimer stop success(this=0x7f594f2260c0, thread_id=5765)
[2024-07-16 11:43:16.180994] INFO ob_timer.cpp:107 [5630][0][Y0-0000000000000000] [lt=28] [dc=0] ObTimer destroy(this=0x7f594f2260c0, thread_id=5765)
[2024-07-16 11:43:16.181033] INFO ob_timer.cpp:366 [5765][266][Y0-0000000000000000] [lt=7] [dc=0] timer thread exit(this=0x7f594f2260c0, thread_id=5765)
[2024-07-16 11:43:16.181111] INFO ob_page_manager.cpp:43 [5765][0][Y0-0000000000000000] [lt=6] [dc=0] unregister pm finish(&pm=0x7f593b98a900, pm.get_tid()=5765)
[2024-07-16 11:43:16.181916] INFO thread_mgr.cpp:72 [5630][0][Y0-0000000000000000] [lt=6] [dc=0] destroy tg(tg_id=259, tg=0x7f594f2260a0, tg->attr_={name:“LogDiskMon”, desc:"", scope:2, type:3})
[2024-07-16 11:43:16.191707] INFO [STORAGE] ob_store_file_system.cpp:855 [5630][0][Y0-0000000000000000] [lt=9] [dc=0] destroy store file system
[2024-07-16 11:43:16.191741] INFO [COMMON] ob_kv_storecache.cpp:541 [5630][0][Y0-0000000000000000] [lt=30] [dc=0] Success to deregister cache, (cache_id=2)
[2024-07-16 11:43:16.191749] INFO [COMMON] ob_kv_storecache.cpp:541 [5630][0][Y0-0000000000000000] [lt=5] [dc=0] Success to deregister cache, (cache_id=3)
[2024-07-16 11:43:16.191751] INFO [COMMON] ob_kv_storecache.cpp:541 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] Success to deregister cache, (cache_id=4)
[2024-07-16 11:43:16.191759] INFO [COMMON] ob_kv_storecache.cpp:541 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] Success to deregister cache, (cache_id=5)
[2024-07-16 11:43:16.195968] INFO [COMMON] ob_kv_storecache.cpp:541 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] Success to deregister cache, (cache_id=6)
[2024-07-16 11:43:16.196001] INFO [CLOG] ob_clog_aggre_runnable.cpp:122 [5630][0][Y0-0000000000000000] [lt=27] [dc=0] clog aggre runnable stop success(this=0xdea89c0)
[2024-07-16 11:43:16.196004] INFO [CLOG] ob_clog_aggre_runnable.cpp:127 [5630][0][Y0-0000000000000000] [lt=3] [dc=0] clog aggre runnable wait success(this=0xdea89c0)
[2024-07-16 11:43:16.196082] INFO [CLOG] ob_clog_aggre_runnable.cpp:143 [5630][0][Y0-0000000000000000] [lt=2] [dc=0] clog aggre runnable destroy(this=0xdea89c0)
[2024-07-16 11:43:16.196223] INFO thread_mgr.cpp:72 [5630][0][Y0-0000000000000000] [lt=15] [dc=0] destroy tg(tg_id=67, tg=0x7f59a10c10f0, tg->attr_={name:“HAGtsMgr”, desc:"", scope:1, type:3})
[2024-07-16 11:43:16.196248] INFO ob_timer.cpp:84 [5630][0][Y0-0000000000000000] [lt=21] [dc=0] ObTimer stop success(this=0x7f59a10c1110, thread_id=5813)
[2024-07-16 11:43:16.196256] INFO ob_timer.cpp:107 [5630][0][Y0-0000000000000000] [lt=4] [dc=0] ObTimer destroy(this=0x7f59a10c1110, thread_id=5813)
[2024-07-16 11:43:16.196273] INFO ob_timer.cpp:366 [5813][360][Y0-0000000000000000] [lt=3] [dc=0] timer thread exit(this=0x7f59a10c1110, thread_id=5813)
[2024-07-16 11:43:16.196470] INFO ob_page_manager.cpp:43 [5813][0][Y0-0000000000000000] [lt=4] [dc=0] unregister pm finish(&pm=0x7f5901617900, pm.get_tid()=5813)
[2024-07-16 11:43:16.197286] INFO thread_mgr.cpp:72 [5630][0][Y0-0000000000000000] [lt=3] [dc=0] destroy tg(tg_id=68, tg=0x7f59a10c1530, tg->attr_={name:“HAGtsHB”, desc:"", scope:1, type:3})
[2024-07-16 11:43:16.197304] INFO ob_timer.cpp:84 [5630][0][Y0-0000000000000000] [lt=16] [dc=0] ObTimer stop success(this=0x7f59a10c1550, thread_id=5814)
[2024-07-16 11:43:16.197309] INFO ob_timer.cpp:107 [5630][0][Y0-0000000000000000] [lt=3] [dc=0] ObTimer destroy(this=0x7f59a10c1550, thread_id=5814)
[2024-07-16 11:43:16.197354] INFO ob_timer.cpp:366 [5814][362][Y0-0000000000000000] [lt=4] [dc=0] timer thread exit(this=0x7f59a10c1550, thread_id=5814)
[2024-07-16 11:43:16.197410] INFO ob_page_manager.cpp:43 [5814][0][Y0-0000000000000000] [lt=4] [dc=0] unregister pm finish(&pm=0x7f59011f1900, pm.get_tid()=5814)
[2024-07-16 11:43:16.198125] INFO [STORAGE] ob_ha_gts_mgr.cpp:258 [5630][0][Y0-0000000000000000] [lt=2] [dc=0] ObHaGtsManager destroy
[2024-07-16 11:43:16.198161] INFO ob_timer.cpp:107 [5630][0][Y0-0000000000000000] [lt=16] [dc=0] ObTimer destroy(this=0xdea9d58, thread_id=5815)
[2024-07-16 11:43:16.198190] INFO ob_timer.cpp:366 [5815][364][Y0-0000000000000000] [lt=5] [dc=0] timer thread exit(this=0xdea9d58, thread_id=5815)
[2024-07-16 11:43:16.198238] INFO ob_page_manager.cpp:43 [5815][0][Y0-0000000000000000] [lt=2] [dc=0] unregister pm finish(&pm=0x7f59005cb900, pm.get_tid()=5815)
[2024-07-16 11:43:16.198813] INFO [STORAGE] ob_ha_gts_source.cpp:289 [5630][0][Y0-0000000000000000] [lt=3] [dc=0] ObHaGtsSource destroy
[2024-07-16 11:43:16.198834] INFO [STORAGE] ob_auto_part_scheduler.cpp:67 [5630][0][Y0-0000000000000000] [lt=15] [dc=0] ObAutoPartScheduler stop finished
[2024-07-16 11:43:16.198837] INFO [STORAGE] ob_auto_part_scheduler.cpp:73 [5630][0][Y0-0000000000000000] [lt=3] [dc=0] ObAutoPartScheduler wait finished
[2024-07-16 11:43:16.198839] INFO [STORAGE] ob_auto_part_scheduler.cpp:82 [5630][0][Y0-0000000000000000] [lt=2] [dc=0] ObAutoPartScheduler destroy
[2024-07-16 11:43:16.198956] WARN [SERVER] destroy (ob_server.cpp:422) [5630][0][Y0-0000000000000000] [lt=1] [dc=0] partition service destroyed
[2024-07-16 11:43:16.198972] INFO [STORAGE.TRANS] ob_gc_partition_adapter.cpp:112 [5630][0][Y0-0000000000000000] [lt=15] [dc=0] ObGCPartitionAdapter destroyed
[2024-07-16 11:43:16.198975] WARN [SERVER] destroy (ob_server.cpp:424) [5630][0][Y0-0000000000000000] [lt=3] [dc=0] gc partition adapter destroyed
[2024-07-16 11:43:16.198991] WARN [SERVER] destroy (ob_server.cpp:426) [5630][0][Y0-0000000000000000] [lt=2] [dc=0] location cache destroyed
[2024-07-16 11:43:16.199014] INFO [STORAGE.TRANS] ob_weak_read_service.cpp:50 [5630][0][Y0-0000000000000000] [lt=22] [dc=0] [WRS] weak read service begin destroy
[2024-07-16 11:43:16.199016] INFO [STORAGE.TRANS] ob_weak_read_service.cpp:73 [5630][0][Y0-0000000000000000] [lt=2] [dc=0] [WRS] weak read service thread stop
[2024-07-16 11:43:16.199021] INFO [STORAGE.TRANS] ob_weak_read_service.cpp:79 [5630][0][Y0-0000000000000000] [lt=2] [dc=0] [WRS] weak read service thread wait
[2024-07-16 11:43:16.199022] INFO [STORAGE.TRANS] ob_weak_read_service.cpp:57 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] [WRS] weak read service destroy succ
[2024-07-16 11:43:16.199024] WARN [SERVER] destroy (ob_server.cpp:428) [5630][0][Y0-0000000000000000] [lt=1] [dc=0] weak read service destroyed
[2024-07-16 11:43:16.199027] INFO easy_io.c:249 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] easy_io_wait join monitor, tp=0x7f5998d49dc0 tid=0
[2024-07-16 11:43:16.199029] INFO easy_io.c:249 [5630][0][Y0-0000000000000000] [lt=2] [dc=0] easy_io_wait join monitor, tp=0x7f5998dd3490 tid=0
[2024-07-16 11:43:16.199031] INFO easy_io.c:249 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] easy_io_wait join monitor, tp=0x7f5998e626b0 tid=0
[2024-07-16 11:43:16.199033] INFO easy_io.c:249 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] easy_io_wait join monitor, tp=0x7f5998e61360 tid=0
[2024-07-16 11:43:16.199116] WARN [SERVER] destroy (ob_server.cpp:430) [5630][0][Y0-0000000000000000] [lt=2] [dc=0] net frame destroyed
[2024-07-16 11:43:16.199137] WARN [SERVER] destroy (ob_server.cpp:433) [5630][0][Y0-0000000000000000] [lt=13] [dc=0] oss storage destroyed
[2024-07-16 11:43:16.199141] INFO [COMMON] ob_io_manager.cpp:468 [5630][0][Y0-0000000000000000] [lt=3] [dc=0] start destroy io manager
[2024-07-16 11:43:16.243713] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=7] [dc=0] clock generator not inited
[2024-07-16 11:43:16.243967] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=244] [dc=0] clock generator not inited
[2024-07-16 11:43:16.244000] WARN [SHARE.SCHEMA] check_formal_guard (ob_schema_getter_guard.cpp:7125) [5808][350][Y0-0000000000000000] [lt=26] [dc=0] local schema_version is not formal, try again(ret=-5627, schema_version=1)
[2024-07-16 11:43:16.244022] WARN [STORAGE] runTimerTask (ob_build_index_scheduler.cpp:1649) [5808][350][Y0-0000000000000000] [lt=19] [dc=0] schema_guard is not formal(ret=-5627)
[2024-07-16 11:43:16.244009] INFO ob_page_manager.cpp:43 [5645][0][Y0-0000000000000000] [lt=6] [dc=0] unregister pm finish(&pm=0x7f5988051900, pm.get_tid()=5645)
[2024-07-16 11:43:16.247666] INFO [COMMON] ob_io_disk.cpp:711 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] io_fault_detector_ stopped
[2024-07-16 11:43:16.247690] INFO thread_mgr.cpp:72 [5630][0][Y0-0000000000000000] [lt=15] [dc=0] destroy tg(tg_id=0, tg=0x7f59a10db770, tg->attr_={name:“test1”, desc:"", scope:1, type:3})
[2024-07-16 11:43:16.273911] WARN [COMMON] get_warning_disks (ob_io_disk.cpp:930) [5650][38][Y0-0000000000000000] [lt=8] [dc=0] not init(ret=-4006)
[2024-07-16 11:43:16.273961] WARN [COMMON] check_disk_error (ob_io_manager.cpp:699) [5650][38][Y0-0000000000000000] [lt=48] [dc=0] fail to get warning disks(ret=-4006)
[2024-07-16 11:43:16.274088] INFO ob_page_manager.cpp:43 [5650][0][Y0-0000000000000000] [lt=2] [dc=0] unregister pm finish(&pm=0x7f59868ad900, pm.get_tid()=5650)
[2024-07-16 11:43:16.362939] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=13] [dc=0] clock generator not inited
[2024-07-16 11:43:16.362968] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=28] [dc=0] clock generator not inited
[2024-07-16 11:43:16.362975] WARN [SHARE.SCHEMA] check_formal_guard (ob_schema_getter_guard.cpp:7125) [5808][350][Y0-0000000000000000] [lt=2] [dc=0] local schema_version is not formal, try again(ret=-5627, schema_version=1)
[2024-07-16 11:43:16.362981] WARN [STORAGE] runTimerTask (ob_build_index_scheduler.cpp:1649) [5808][350][Y0-0000000000000000] [lt=5] [dc=0] schema_guard is not formal(ret=-5627)
[2024-07-16 11:43:16.386039] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:601) [5681][100][Y0-0000000000000000] [lt=2] [dc=0] tenant manager not init(ret=-4006)
[2024-07-16 11:43:16.386056] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [5681][100][Y0-0000000000000000] [lt=16] [dc=0] Fail to get all tenant ids, (ret=-4006)
[2024-07-16 11:43:16.386065] INFO [COMMON] ob_kvcache_store.cpp:799 [5681][100][Y0-0000000000000000] [lt=3] [dc=0] Wash compute wash size(sys_total_wash_size=-5604429005, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[])
[2024-07-16 11:43:16.386072] INFO [COMMON] ob_kvcache_store.cpp:323 [5681][100][Y0-0000000000000000] [lt=3] [dc=0] Wash time detail, (refresh_score_time=54, compute_wash_size_time=39, wash_sort_time=1, wash_time=0)
[2024-07-16 11:43:16.463512] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=3] [dc=0] clock generator not inited
[2024-07-16 11:43:16.463539] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=26] [dc=0] clock generator not inited
[2024-07-16 11:43:16.463543] WARN [SHARE.SCHEMA] check_formal_guard (ob_schema_getter_guard.cpp:7125) [5808][350][Y0-0000000000000000] [lt=2] [dc=0] local schema_version is not formal, try again(ret=-5627, schema_version=1)
[2024-07-16 11:43:16.463549] WARN [STORAGE] runTimerTask (ob_build_index_scheduler.cpp:1649) [5808][350][Y0-0000000000000000] [lt=5] [dc=0] schema_guard is not formal(ret=-5627)
[2024-07-16 11:43:16.563772] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=3] [dc=0] clock generator not inited
[2024-07-16 11:43:16.563795] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=22] [dc=0] clock generator not inited
[2024-07-16 11:43:16.563798] WARN [SHARE.SCHEMA] check_formal_guard (ob_schema_getter_guard.cpp:7125) [5808][350][Y0-0000000000000000] [lt=1] [dc=0] local schema_version is not formal, try again(ret=-5627, schema_version=1)
[2024-07-16 11:43:16.563804] WARN [STORAGE] runTimerTask (ob_build_index_scheduler.cpp:1649) [5808][350][Y0-0000000000000000] [lt=5] [dc=0] schema_guard is not formal(ret=-5627)
[2024-07-16 11:43:16.570483] INFO ob_page_manager.cpp:43 [5647][0][Y0-0000000000000000] [lt=10] [dc=0] unregister pm finish(&pm=0x7f5987541900, pm.get_tid()=5647)
[2024-07-16 11:43:16.570928] INFO ob_page_manager.cpp:43 [5646][0][Y0-0000000000000000] [lt=8] [dc=0] unregister pm finish(&pm=0x7f5987767900, pm.get_tid()=5646)
[2024-07-16 11:43:16.571044] INFO ob_page_manager.cpp:43 [5648][0][Y0-0000000000000000] [lt=2] [dc=0] unregister pm finish(&pm=0x7f598731b900, pm.get_tid()=5648)
[2024-07-16 11:43:16.571088] INFO ob_page_manager.cpp:43 [5649][0][Y0-0000000000000000] [lt=4] [dc=0] unregister pm finish(&pm=0x7f5986cf5900, pm.get_tid()=5649)
[2024-07-16 11:43:16.575939] INFO [COMMON] ob_io_manager.cpp:480 [5630][0][Y0-0000000000000000] [lt=4] [dc=0] finish destroy io manager
[2024-07-16 11:43:16.575957] WARN [SERVER] destroy (ob_server.cpp:436) [5630][0][Y0-0000000000000000] [lt=17] [dc=0] io manager destroyed
[2024-07-16 11:43:16.598874] WARN [COMMON] get_all_tenant_id (ob_tenant_mgr.cpp:601) [5681][100][Y0-0000000000000000] [lt=2] [dc=0] tenant manager not init(ret=-4006)
[2024-07-16 11:43:16.598899] WARN [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:659) [5681][100][Y0-0000000000000000] [lt=24] [dc=0] Fail to get all tenant ids, (ret=-4006)
[2024-07-16 11:43:16.598914] INFO [COMMON] ob_kvcache_store.cpp:799 [5681][100][Y0-0000000000000000] [lt=4] [dc=0] Wash compute wash size(sys_total_wash_size=-5707189453, global_cache_size=0, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[])
[2024-07-16 11:43:16.598926] INFO [COMMON] ob_kvcache_store.cpp:323 [5681][100][Y0-0000000000000000] [lt=5] [dc=0] Wash time detail, (refresh_score_time=87, compute_wash_size_time=58, wash_sort_time=3, wash_time=0)
[2024-07-16 11:43:16.675251] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=2] [dc=0] clock generator not inited
[2024-07-16 11:43:16.675366] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:65) [5808][350][Y0-0000000000000000] [lt=97] [dc=0] clock generator not inited
[2024-07-16 11:43:16.675374] WARN [SHARE.SCHEMA] check_formal_guard (ob_schema_getter_guard.cpp:7125) [5808][350][Y0-0000000000000000] [lt=2] [dc=0] local schema_version is not formal, try again(ret=-5627, schema_version=1)
[2024-07-16 11:43:16.675395] INFO ob_page_manager.cpp:43 [5651][0][Y0-0000000000000000] [lt=8] [dc=0] unregister pm finish(&pm=0x7f5985887900, pm.get_tid()=5651)
[2024-07-16 11:43:16.675402] WARN [STORAGE] runTimerTask (ob_build_index_scheduler.cpp:1649) [5808][350][Y0-0000000000000000] [lt=27] [dc=0] schema_guard is not formal(ret=-5627)
[2024-07-16 11:43:16.676593] WARN [SERVER] destroy (ob_server.cpp:438) [5630][0][Y0-0000000000000000] [lt=2] [dc=0] memory dump destroyed
[2024-07-16 11:43:16.676627] WARN [SERVER] destroy (ob_server.cpp:440) [5630][0][Y0-0000000000000000] [lt=17] [dc=0] tenant timezone manager destroyed
[2024-07-16 11:43:16.677019] INFO ob_page_manager.cpp:43 [5637][0][Y0-0000000000000000] [lt=0] [dc=0] unregister pm finish(&pm=0x7f5993494900, pm.get_tid()=5637)
[2024-07-16 11:43:16.677651] WARN [SERVER] destroy (ob_server.cpp:442) [5630][0][Y0-0000000000000000] [lt=3] [dc=0] log compressor destroyed
[2024-07-16 11:43:16.677669] WARN [SERVER] destroy (ob_server.cpp:443) [5630][0][Y0-0000000000000000] [lt=17] [dc=0] destroy observer end
[2024-07-16 11:43:16.677672] ERROR [SERVER] main (main.cpp:494) [5630][0][Y0-0000000000000000] [lt=2] [dc=0] observer init fail(ret=-4002) BACKTRACE:0x9b8f29e 0x99624d1 0x22f0af4 0x22f05db 0x22f0341 0x22eefd8 0x22ed44a 0x7f59a6734555 0x22ebe29
[2024-07-16 11:43:16.677691] INFO [COMMON] ob_kv_storecache.cpp:215 [5630][0][Y0-0000000000000000] [lt=19] [dc=0] Begin destroy the ObKVGlobalCache!
[2024-07-16 11:43:16.677702] INFO ob_timer.cpp:234 [5630][0][Y0-0000000000000000] [lt=2] [dc=0] cancel task(this=0x7f59a10d1510, thread_id=5681, pos=0, wakeup_time_=1105478073, tasks_num_=0, task=tasktype:N9oceanbase6common15ObKVGlobalCache15KVStoreWashTaskE)
[2024-07-16 11:43:16.677707] INFO ob_timer.cpp:234 [5630][0][Y0-0000000000000000] [lt=3] [dc=0] cancel task(this=0x7f59a10d1950, thread_id=5682, pos=0, wakeup_time_=1105441852, tasks_num_=0, task=tasktype:N9oceanbase6common15ObKVGlobalCache16KVMapReplaceTaskE)
[2024-07-16 11:43:16.677709] INFO ob_timer.cpp:107 [5630][0][Y0-0000000000000000] [lt=1] [dc=0] ObTimer destroy(this=0x7f59a10d1510, thread_id=5681)
[2024-07-16 11:43:16.677799] INFO ob_timer.cpp:366 [5681][100][Y0-0000000000000000] [lt=2] [dc=0] timer thread exit(this=0x7f59a10d1510, thread_id=5681)
[2024-07-16 11:43:16.677852] INFO ob_page_manager.cpp:43 [5681][0][Y0-0000000000000000] [lt=4] [dc=0] unregister pm finish(&pm=0x7f5966b69900, pm.get_tid()=5681)
[2024-07-16 11:43:16.697801] INFO ob_timer.cpp:107 [5630][0][Y0-0000000000000000] [lt=2] [dc=0] ObTimer destroy(this=0x7f59a10d1950, thread_id=5682)
[2024-07-16 11:43:16.697839] INFO ob_timer.cpp:366 [5682][102][Y0-0000000000000000] [lt=14] [dc=0] timer thread exit(this=0x7f59a10d1950, thread_id=5682)
[2024-07-16 11:43:16.697897] INFO ob_page_manager.cpp:43 [5682][0][Y0-0000000000000000] [lt=4] [dc=0] unregister pm finish(&pm=0x7f5966743900, pm.get_tid()=5682)
[2024-07-16 11:43:16.768693] INFO [COMMON] ob_kv_storecache.cpp:233 [5630][0][Y0-0000000000000000] [lt=19] [dc=0] The ObKVGlobalCache has been destroyed!
[2024-07-16 11:43:16.768762] INFO [SERVER] main.cpp:508 [5630][0][Y0-0000000000000000] [lt=31] [dc=0] observer is exit(observer_version=“OceanBase CE 3.1.5”)