应用程序报错 No memory or reach tenant memory limit

AntTech_RWNFTN · 2026 年7 月 1 日 16:16

生产环境多次报No memory or reach tenant memory limit，
使用脚本SELECT
zone,
concat( SVR_IP,’:’, SVR_PORT ) observer,
cpu_capacity_max cpu_total,
cpu_assigned_max cpu_assigned,
cpu_capacity - cpu_assigned_max AS cpu_free,
round( memory_limit / 1024 / 1024 / 1024, 2 ) AS memory_total,
round(( memory_limit - mem_capacity )/ 1024 / 1024 / 1024, 2 ) AS system_memory,
round( mem_assigned / 1024 / 1024 / 1024, 2 ) AS mem_assigned,
round(( mem_capacity - mem_assigned )/ 1024 / 1024 / 1024, 2 ) AS memory_free,
round( log_disk_capacity / 1024 / 1024 / 1024, 2 ) AS log_disk_capacity,
round( log_disk_assigned / 1024 / 1024 / 1024, 2 ) AS log_disk_assigned,
round(( log_disk_capacity - log_disk_assigned )/ 1024 / 1024 / 1024, 2 ) AS log_disk_free,
round(( data_disk_capacity / 1024 / 1024 / 1024 ), 2 ) AS data_disk,
round(( data_disk_in_use / 1024 / 1024 / 1024 ), 2 ) AS data_disk_used,
round(( data_disk_capacity - data_disk_in_use )/ 1024 / 1024 / 1024, 2 ) AS data_disk_free
FROM
oceanbase.gv$ob_servers;
查询后结果为

其中mem_assigned 是18.25G 这个值是怎么来的？
再用
SELECT
t.tenant_id,
t.tenant_name,
p.name AS pool_name,
p.zone_list,
p.unit_count,
u.name AS unit_config_name,
u.memory_size / 1024 / 1024 / 1024 AS memory_size_gb,
p.unit_count * u.memory_size / 1024 / 1024 / 1024 AS pool_memory_gb
FROM oceanbase.dba_ob_tenants t
JOIN oceanbase.dba_ob_resource_pools p
ON t.tenant_id = p.tenant_id
JOIN oceanbase.dba_ob_unit_configs u
ON p.unit_config_id = u.unit_config_id;
查询发现结果为

tenant_id	tenant_name	pool_name	zone_list	unit_count	unit_config_name	memory_size_gb	pool_memory_gb
1	sys	sys_pool	zone1	1	sys_unit_config	2	2
1002	test	test_pool	zone1	1	test_unit	16.253857421688	16.253857421688

其中test租户为16.2G 这个memory_size 是怎么计算出来的？

淇铭 · 2026 年7 月 1 日 17:08

信息查一下
grep ‘[OOPS]’ observer.log*
注意[的反斜杠
– Per-tenant hold vs free on each node
SELECT tenant_id, svr_ip, svr_port, hold, free
FROM GV$OB_TENANT_MEMORY
WHERE tenant_id = <your_tenant_id>;

– Unit memory spec
SELECT tenant_id, svr_ip, memory_size
FROM GV$OB_UNITS
WHERE tenant_id = <your_tenant_id>;

SELECT tenant_id, svr_ip, ctx_name, mod_name,
hold / 1024 / 1024 AS hold_mb,
used / 1024 / 1024 AS used_mb
FROM GV$OB_MEMORY
WHERE tenant_id = <your_tenant_id>
ORDER BY hold DESC
LIMIT 20;

SELECT tenant_id, svr_ip,
active_memstore_used / 1024 / 1024 / 1024 AS active_gb,
total_memstore_used / 1024 / 1024 / 1024 AS total_gb,
memstore_limit / 1024 / 1024 / 1024 AS limit_gb
FROM GV$OB_MEMSTORE
WHERE tenant_id = <your_tenant_id>;

AntTech_RWNFTN · 2026 年7 月 1 日 17:37

结果为

tenant_id	svr_ip	svr_port	hold	free
1002	127.0.0.1	2882	5092933632	10614268232

tenant_id	svr_ip	memory_size
1002	127.0.0.1	15707201864

tenant_id	svr_ip	ctx_name	mod_name	hold_mb	used_mb
1002	127.0.0.1	MEMSTORE_CTX_ID	Memstore	1567.65625	1566.939011
1002	127.0.0.1	PLAN_CACHE_CTX_ID	SqlPhyPlan	299.1546021	271.5412102
1002	127.0.0.1	DEFAULT_CTX_ID	MysqlRequesReco	289.40625	289.0236816
1002	127.0.0.1	DEFAULT_CTX_ID	MinorMergeMgr	178.6640625	158.8125
1002	127.0.0.1	CO_STACK	CoStack	172.453125	172.1940308
1002	127.0.0.1	DEFAULT_CTX_ID	BloomFilter	160.0625	160.0000305
1002	127.0.0.1	TX_CALLBACK_CTX_ID	MemtableCallbac	90.9375	88.09570313
1002	127.0.0.1	DEFAULT_CTX_ID	LogGroupBuffer	80.125	80
1002	127.0.0.1	KVSTORE_CACHE_ID	KvstorCacheMb	62	0
1002	127.0.0.1	DEFAULT_CTX_ID	MajorMergeMgr	60.9609375	54.1875
1002	127.0.0.1	DEFAULT_CTX_ID	MrgMemPoolChk	48.1875	48.00034332
1002	127.0.0.1	DEFAULT_CTX_ID	ServerObjecPool	45.9453125	45.8125
1002	127.0.0.1	DEFAULT_CTX_ID	HashBuckDmId	36.8203125	36.75021362
1002	127.0.0.1	DEFAULT_CTX_ID	SqlPlanMon	36.7578125	34.63793945
1002	127.0.0.1	DEFAULT_CTX_ID	HashBuckDTLINT	36.0625	36.00011444
1002	127.0.0.1	DEFAULT_CTX_ID	HashBuckDmChe	36.0625	36.00011444
1002	127.0.0.1	DEFAULT_CTX_ID	DiagnosticInfos	32.25	32.00097656
1002	127.0.0.1	PLAN_CACHE_CTX_ID	SqlPlanCache	29.4609375	26.43587494
1002	127.0.0.1	META_OBJ_CTX_ID	L_TabletPool	28.125	25.02098083
1002	127.0.0.1	META_OBJ_CTX_ID	N_TabletPool	25.4375	25.33813477

SVR_IP	SVR_PORT	TENANT_ID	ACTIVE_SPAN	FREEZE_TRIGGER	FREEZE_CNT	MEMSTORE_USED	MEMSTORE_LIMIT
127.0.0.1	2882	1002	1660944384	2006942772	0	1660944384	7853600900

淇铭 · 2026 年7 月 1 日 17:54

日志中看看是否有这个 Observer 日志中 No memory or reach tenant memory limit 或者搜一下-4013这个信息
从查询的信息看没有内存爆的问题

AntTech_RWNFTN · 2026 年7 月 1 日 18:08

7.0.0.1:2882", request_id=3)

[2026-07-01 00:44:54.390294] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [780][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret=“OB_NEED_WAIT”)

[2026-07-01 00:44:54.395707] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=13][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=10, len=185, pos=153)

[2026-07-01 00:44:54.395892] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=31][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=19, len=364, pos=358)

[2026-07-01 00:44:54.395979] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=19][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=18, len=342, pos=332)

[2026-07-01 00:44:54.396062] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=14][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=18, len=342, pos=332)

[2026-07-01 00:44:54.396143] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=14][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=14, len=265, pos=244)

[2026-07-01 00:44:54.396226] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=13][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=17, len=325, pos=310)

[2026-07-01 00:44:54.396311] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=12][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=18, len=351, pos=331)

[2026-07-01 00:44:54.396389] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=13][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=18, len=327, pos=324)

[2026-07-01 00:44:54.396458] WDIAG [RPC.OBMYSQL] serialize (ob_mysql_row.cpp:54) [2182793][T1002_L0_G0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=13][errcode=-4019] failed to encode cell(ret=-4019, cell_idx=18, len=354, pos=331)

[2026-07-01 00:44:54.401094] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [491][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1})

[2026-07-01 00:44:54.414567] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:763) [89][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] Cache replace map node details(ret=0, replace_node_count=47, replace_time=0, replace_start_pos=503312, replace_num=31457)

[2026-07-01 00:44:54.414594] INFO [COMMON] replace_map (ob_kv_storecache.cpp:694) [89][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=47, map_once_replace_num_=31457, map_replace_skip_count_=0)

[2026-07-01 00:44:54.417025] INFO [SHARE] run_loop_ (ob_bg_thread_monitor.cpp:326) [932][BGThreadMonitor][T0][Y0-0000000000000000-0-0] [lt=11] current monitor number(seq_=-1)

[2026-07-01 00:44:54.417830] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:100) [258][TsMgr][T1][Y0-0000000000000000-0-0] [lt=4] is logging

[2026-07-01 00:44:54.417853] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:108) [258][TsMgr][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] submit log callback use too mush time(submit_log_ts=1782866694263333, cb={log_ts:{val:1782866694262681002, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1782866706261672000}, service_type=0, latest_log_ts={val:1782866686261672000, v:0}, self=“127.0.0.1:2882”)

[2026-07-01 00:44:54.417881] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:48) [258][TsMgr][T1][Y0-0000000000000000-0-0] [lt=25] submit log with lock(ret=-4023, service_type_=0, last_id=1782866706261672000, limited_id=1782866706261672000)

[2026-07-01 00:44:54.417913] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [258][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=0, ret=“OB_SUCCESS”, tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1782866694417818], gts:1782866694417692000, latest_srr:[mts=1782866694417818]})

[2026-07-01 00:44:54.417933] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:100) [258][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=15] is logging

[2026-07-01 00:44:54.417942] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:108) [258][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] submit log callback use too mush time(submit_log_ts=1782866693514908, cb={log_ts:{val:1782866693514162001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1782866705232866000}, service_type=0, latest_log_ts={val:1782866685232866000, v:0}, self=“127.0.0.1:2882”)

[2026-07-01 00:44:54.417939] INFO [STORAGE.TABLELOCK] get_data_version_ (ob_table_lock_common.cpp:476) [248][TSWorker0][T1][YB427F000001-000644D3C99DCB1C-0-0] [lt=13] refresh the data version because of tenant changed(tenant_id=1, last_tenant_id=1001)

[2026-07-01 00:44:54.417954] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:48) [258][TsMgr][T1001][Y0-0000000000000000-0-0] [lt=13] submit log with lock(ret=-4023, service_type_=0, last_id=1782866705232866000, limited_id=1782866705232866000)

[2026-07-01 00:44:54.417969] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [258][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts(ret=0, ret=“OB_SUCCESS”, tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1782866694417929], gts:1782866694417692000, latest_srr:[mts=1782866694417929]})

[2026-07-01 00:44:54.418259] INFO [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:100) [258][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10] is logging

[2026-07-01 00:44:54.418270] WDIAG [STORAGE.TRANS] submit_log_ (ob_id_service.cpp:108) [258][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] submit log callback use too mush time(submit_log_ts=1782866692916264, cb={log_ts:{val:1782866692915480001, v:0}, log_buf:"", pos:40, id_srv_type:0, limited_id:1782866704743703000}, service_type=0, latest_log_ts={val:1782866684743703000, v:0}, self=“127.0.0.1:2882”)

[2026-07-01 00:44:54.418283] INFO [STORAGE.TRANS] submit_log_with_lock_ (ob_id_service.cpp:48) [258][TsMgr][T1002][Y0-0000000000000000-0-0] [lt=12] submit log with lock(ret=-4023, service_type_=0, last_id=1782866704743703000, limited_id=1782866704743703000)

[2026-07-01 00:44:54.418301] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [258][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] refresh gts(ret=0, ret=“OB_SUCCESS”, tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1782866694418255], gts:1782866694417692000, latest_srr:[mts=1782866694418255]})

[2026-07-01 00:44:54.419328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [349][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR=“cgroup”, ret=-4027)

[2026-07-01 00:44:54.422948] INFO [LIB] stat (utility.h:1219) [736][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=5] [PALF STAT WRITE LOG COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=1543965, this=0xfffb74fd87f0, extra_info="")

[2026-07-01 00:44:54.422982] WDIAG [PALF] inner_append_log (palf_handle_impl.cpp:2236) [736][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4389] write log cost too much time(ret=-4389, this={palf_id:1, self:“127.0.0.1:2882”, has_set_deleted:false}, lsn_array=[{lsn:66394091914}], scn_array=[{val:1782866684743703001, v:0}], curr_size=122, accum_size=122, time_cost=1543965)

[2026-07-01 00:44:54.423007] INFO [PALF] inner_append_log (palf_handle_impl.cpp:2239) [736][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT INNER APPEND LOG SIZE](this={palf_id:1, self:“127.0.0.1:2882”, has_set_deleted:false}, accum_size=122)

[2026-07-01 00:44:54.423026] INFO [LIB] stat (utility.h:1236) [736][T1002_IOWorker][T1002][Y0-0000000000000000-0-0] [lt=7] [PALF STAT IO TASK IN QUEUE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=9, this=0xfffb78bda6b0, extra_info="")

[2026-07-01 00:44:54.423047] WDIAG [PALF] try_update_match_lsn_map_ (log_sliding_window.cpp:4109) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=0] [MATCH LSN ADVANCE DELAY]match_lsn advance delay too much time(ret=0, palf_id=1, self=“127.0.0.1:2882”, server=“127.0.0.1:2882”, update_func={old_end_lsn:{lsn:66394091914}, new_end_lsn:{lsn:66394092036}, old_advance_time_us:1782866692878894, new_ack_time_us:1782866694423041, advance delay(us):1544147})

[2026-07-01 00:44:54.423086] INFO [PALF] try_advance_committed_lsn_ (log_sliding_window.cpp:1758) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT COMMITTED LOG SIZE](palf_id=1, self=“127.0.0.1:2882”, committed size=122)

[2026-07-01 00:44:54.423127] INFO [LIB] stat (utility.h:1219) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=14] [PALF STAT PUSH LOG TO FOLLOWERS RPC POST COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=2, this=0xfffb74fcc130, extra_info="")

[2026-07-01 00:44:54.423153] INFO [PALF] handle_next_submit_log_ (log_sliding_window.cpp:1080) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT GROUP LOG INFO](palf_id=1, self=“127.0.0.1:2882”, role=“LEADER”, total_group_log_cnt=1, avg_log_batch_cnt=5, total_group_log_size=2107, avg_group_log_size=2107)

[2026-07-01 00:44:54.423191] INFO [LIB] stat (utility.h:1219) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=13] [PALF STAT FS CB EXCUTE COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=17, this=0xfffb74fcc360, extra_info=“palf_id:1”)

[2026-07-01 00:44:54.423201] INFO [LIB] stat (utility.h:1219) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF STAT LOG LIFE TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=6358202, this=0xfffb74fcc590, extra_info=“palf_id:1”)

[2026-07-01 00:44:54.423212] INFO [PALF] sliding_cb (log_sliding_window.cpp:2278) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF STAT LOG TASK TIME](palf_id=1, self=“127.0.0.1:2882”, total_slide_log_cnt=1, avg_log_gen_to_freeze_time=4813963, avg_log_gen_to_submit_time=4813994, avg_log_submit_to_flush_time=1544071, avg_log_submit_to_slide_time=1544208)

[2026-07-01 00:44:54.423225] WDIAG [PALF] sliding_cb (log_sliding_window.cpp:2289) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4389] log_task life cost too much time(palf_id=1, self=“127.0.0.1:2882”, log_id=268602429, log_task={header:{begin_lsn:{lsn:66394091914}, end_lsn:{lsn:66394092036}, log_id:268602429, min_scn:{val:1782866684743703001, v:0}, max_scn:{val:1782866684743703001, v:0}, data_len:66, proposal_id:6, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:66394091914}, data_checksum:667309872, accum_checksum:699655902, is_padding_log:false, is_raw_write:false}, state_map:{val:62}, ref_cnt:0, gen_ts:1782866688064971, freeze_ts:1782866692878934, submit_ts:1782866692878965, flushed_ts:1782866694423036, gen_to_freeze cost time:4813963, gen_to_submit cost time:4813994, submit_to_flush cost time:1544071}, fs_cb_begin_ts=1782866694423173, log_life_time=6358202)

[2026-07-01 00:44:54.423269] INFO [LIB] stat (utility.h:1219) [735][T1002_LogIOCb0][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT FLUSH CB COST TIME](cur_stat_count=1, stat_interval=1000000, avg (count/cost)=232, this=0xfffb74fd8a20, extra_info="")

[2026-07-01 00:44:54.423251] WDIAG [CLOG] statistics_cb_cost_ (ob_log_apply_service.cpp:943) [809][T1002_ApplySrv2][T1002][YB427F000001-0006450246EE43B1-0-0] [lt=16][errcode=-4389] cb cost too much time(lsn={lsn:66394091970}, scn={val:1782866684743703001, v:0}, idx=25, total_cost_time=6357999, append_cost_time=22, cb_wait_thread_time=6358255, cb_wait_commit_time=0, cb_cost_time=-278, append_start_time=1782866688064967, append_finish_time=1782866688064989, cb_first_handle_time=1782866694423244, cb_finish_time=1782866694422966)

[2026-07-01 00:44:54.423281] INFO [PALF] read (log_cache.cpp:101) [741][T1002_LogShared][T1002][Y0-0000000000000000-0-0] [lt=18] [PALF STAT HOT CACHE HIT RATE](palf_id=1, read_size=122, hit_cnt=1, read_cnt=1, hit rate=1.000000000000000000e+00)

[2026-07-01 00:44:54.444579] WARN [USING_LOG_PREFIX] print_alloc_failed_msg (alloc_failed_reason.cpp:141) [731][T1002_DiskCB0][T1002][YB427F000001-0006450246EE4278-0-0] [lt=16][errcode=-4013] [oops]: alloc failed reason is that server memory has reached the upper limit(server_hold: 8588558336, server_limit: 8589934592, alloc_size: 2097152)

[2026-07-01 00:44:54.444960] WARN [USING_LOG_PREFIX] print_alloc_failed_msg (alloc_failed_reason.cpp:141) [950][T1002_DiskCB1][T1002][YB427F000001-0006450246EE465B-0-0] [lt=15][errcode=-4013] [oops]: alloc failed reason is that server memory has reached the upper limit(server_hold: 8588558336, server_limit: 8589934592, alloc_size: 2097152)

[2026-07-01 00:44:54.451745] INFO [COMMON] reclaim (ob_kvcache_hazard_domain.h:428) [92][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=337] [KV_CACHE_HAZARD_DOMAIN] finish reclaim(retired_memory_size_=957153280, reclaimed_memory_size=22888448)

[2026-07-01 00:44:54.451765] WDIAG [STORAGE] put_cache_block (ob_micro_block_cache.cpp:1399) [731][T1002_DiskCB0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=495][errcode=-4013] fail to transfrom(ret=-4013)

[2026-07-01 00:44:54.451781] WDIAG [STORAGE] process_block (ob_micro_block_cache.cpp:479) [731][T1002_DiskCB0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=14][errcode=-4013] Failed to put block to cache(ret=-4013)

[2026-07-01 00:44:54.451786] WDIAG [STORAGE] process (ob_micro_block_cache.cpp:617) [731][T1002_DiskCB0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=5][errcode=-4013] process_block failed(ret=-4013)

[2026-07-01 00:44:54.451781] WDIAG [STORAGE] transfrom (ob_micro_block_cache.cpp:1153) [950][T1002_DiskCB1][T1002][YB427F000001-0006450246EE423D-0-0] [lt=523][errcode=-4013] Fail to decompress data with preallocated buffer(ret=-4013, header={magic:1005, version:3, header_size:312, header_checksum:13351, column_count:31, rowkey_column_count:3, has_column_checksum:1, row_count:43, row_store_type:1, opt:10, var_column_count:4, compressor_type:4, row_offset:11695, original_length:95827, max_merged_trans_version:1782782536152620000, min_merged_trans_version:281452495395717, data_length:74496, data_zlength:17111, data_checksum:465870949, column_checksums:0xfffac4025b85, single_version_rows:0, contain_uncommitted_rows:1, is_last_row_last_flag:0, is_valid():true})

[2026-07-01 00:44:54.451807] WDIAG [STORAGE] put_cache_block (ob_micro_block_cache.cpp:1399) [950][T1002_DiskCB1][T1002][YB427F000001-0006450246EE423D-0-0] [lt=25][errcode=-4013] fail to transfrom(ret=-4013)

[2026-07-01 00:44:54.451814] WDIAG [STORAGE] process_block (ob_micro_block_cache.cpp:479) [950][T1002_DiskCB1][T1002][YB427F000001-0006450246EE423D-0-0] [lt=6][errcode=-4013] Failed to put block to cache(ret=-4013)

[2026-07-01 00:44:54.451801] WDIAG [COMMON] handle (ob_io_struct.cpp:3085) [731][T1002_DiskCB0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=13][errcode=-4013] fail to callback(ret=-4013, *req={is_inited_:true, tenant_id_:1002, control_block_:0xfffbd4006e58, ref_cnt_:1, raw_buf_:0xfffa16cea270, fd_:{first_id:1152921504613466777, second_id:7852, third_id:0, fd_id:-1, slot_version:-1, device_handle:0xfffbe3193430}, is_object_device_req():false, trace_id_:YB427F000001-000645022CDE941E-0-0, retry_count_:0, tenant_io_mgr_:{ptr:0xfffb7d546030}, storage_accesser:{ptr:null}, io_result_:{is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, complete_size_:20480, offset_:313232, size_:15454, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:“READ”, group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:0, fs_errno_:0}, tenant_id_:1002, tenant_io_mgr_:{ptr:0xfffb7d546030}, user_data_buf_:null, buf_:null, io_callback_:0xfff9156f5b80, time_log:{begin_ts:1782866694395113, enqueue_used:1, dequeue_used:7, submit_used:1, return_used:48966, callback_enqueue_used:2, callback_dequeue_used:5162, callback_finish_used:-1, end_used:-1}}, part_id:-1}, MTL_ID()=1002)

[2026-07-01 00:44:54.451822] WDIAG [STORAGE] process (ob_micro_block_cache.cpp:617) [950][T1002_DiskCB1][T1002][YB427F000001-0006450246EE423D-0-0] [lt=8][errcode=-4013] process_block failed(ret=-4013)

[2026-07-01 00:44:54.451833] WDIAG [COMMON] run1 (ob_io_struct.cpp:2969) [731][T1002_DiskCB0][T1002][YB427F000001-000645022CDE941E-0-0] [lt=27][errcode=-4013] handle request failed(ret=-4013, req={is_inited_:true, tenant_id_:1002, control_block_:0xfffbd4006e58, ref_cnt_:1, raw_buf_:null, fd_:{first_id:1152921504613466777, second_id:7852, third_id:0, fd_id:-1, slot_version:-1, device_handle:0xfffbe3193430}, is_object_device_req():false, trace_id_:YB427F000001-000645022CDE941E-0-0, retry_count_:0, tenant_io_mgr_:{ptr:0xfffb7d546030}, storage_accesser:{ptr:null}, io_result_:{is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:20480, offset_:313232, size_:15454, timeout_us_:10000000, result_ref_cnt_:2, out_ref_cnt_:1, flag_:{mode:“READ”, group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4013, fs_errno_:0}, tenant_id_:1002, tenant_io_mgr_:{ptr:0xfffb7d546030}, user_data_buf_:null, buf_:null, io_callback_:0xfff9156f5b80, time_log:{begin_ts:1782866694395113, enqueue_used:1, dequeue_used:7, submit_used:1, return_used:48966, callback_enqueue_used:2, callback_dequeue_used:5162, callback_finish_used:-1, end_used:1782866694451831}}, part_id:-1})

[2026-07-01 00:44:54.451834] WDIAG [COMMON] handle (ob_io_struct.cpp:3085) [950][T1002_DiskCB1][T1002][YB427F000001-0006450246EE423D-0-0] [lt=11][errcode=-4013] fail to callback(ret=-4013, *req={is_inited_:true, tenant_id_:1002, control_block_:0xfffbd40825d8, ref_cnt_:1, raw_buf_:0xfffac4024250, fd_:{first_id:1152921504613477900, second_id:4387, third_id:0, fd_id:-1, slot_version:-1, device_handle:0xfffbe3193430}, is_object_device_req():false, trace_id_:YB427F000001-0006450246EE423D-0-0, retry_count_:0, tenant_io_mgr_:{ptr:0xfffb7d546030}, storage_accesser:{ptr:null}, io_result_:{is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, complete_size_:20480, offset_:490309, size_:17423, timeout_us_:10000000, result_ref_cnt_:3, out_ref_cnt_:2, flag_:{mode:“READ”, group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:0, fs_errno_:0}, tenant_id_:1002, tenant_io_mgr_:{ptr:0xfffb7d546030}, user_data_buf_:null, buf_:null, io_callback_:0xfffab88e2cc0, time_log:{begin_ts:1782866694438349, enqueue_used:1, dequeue_used:7, submit_used:1, return_used:6790, callback_enqueue_used:3, callback_dequeue_used:4645, callback_finish_used:-1, end_used:-1}}, part_id:-1}, MTL_ID()=1002)

[2026-07-01 00:44:54.451876] WDIAG [COMMON] run1 (ob_io_struct.cpp:2969) [950][T1002_DiskCB1][T1002][YB427F000001-0006450246EE423D-0-0] [lt=33][errcode=-4013] handle request failed(ret=-4013, req={is_inited_:true, tenant_id_:1002, control_block_:0xfffbd40825d8, ref_cnt_:1, raw_buf_:null, fd_:{first_id:1152921504613477900, second_id:4387, third_id:0, fd_id:-1, slot_version:-1, device_handle:0xfffbe3193430}, is_object_device_req():false, trace_id_:YB427F000001-0006450246EE423D-0-0, retry_count_:0, tenant_io_mgr_:{ptr:0xfffb7d546030}, storage_accesser:{ptr:null}, io_result_:{is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:20480, offset_:490309, size_:17423, timeout_us_:10000000, result_ref_cnt_:3, out_ref_cnt_:2, flag_:{mode:“READ”, group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4013, fs_errno_:0}, tenant_id_:1002, tenant_io_mgr_:{ptr:0xfffb7d546030}, user_data_buf_:null, buf_:null, io_callback_:0xfffab88e2cc0, time_log:{begin_ts:1782866694438349, enqueue_used:1, dequeue_used:7, submit_used:1, return_used:6790, callback_enqueue_used:3, callback_dequeue_used:4645, callback_finish_used:-1, end_used:1782866694451872}}, part_id:-1})

[2026-07-01 00:44:54.451883] WDIAG [COMMON] wait (ob_io_define.cpp:1966) [2182941][T1002_L0_G0][T1002][YB427F000001-0006450246EE423D-0-0] [lt=46][errcode=-4013] IO error, (ret=-4013, *result_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, complete_size_:20480, offset_:490309, size_:17423, timeout_us_:10000000, result_ref_cnt_:3, out_ref_cnt_:2, flag_:{mode:“READ”, group_id_:0, func_type_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:false, is_detect_:false, is_write_through_:false, is_sealed_:true, is_time_detect_:false, need_close_dev_and_fd_:false, reserved_:0}, ret_code_:{io_ret_:-4013, fs_errno_:0}, tenant_id_:1002, tenant_io_mgr_:{ptr:0xfffb7d546030}, user_data_buf_:null, buf_:null, io_callback_:0xfffab88e2cc0, time_log:{begin_ts:1782866694438349, enqueue_used:1, dequeue_used:7, submit_used:1, return_used:6790, callback_enqueue_used:3, callback_dequeue_used:4645, callback_finish_used:2107, end_used:-31}})

[2026-07-01 00:44:54.451937] WDIAG [STORAGE] wait (ob_storage_object_handle.cpp:372) [2182941][T1002_L0_G0][T1002][YB427F000001-0006450246EE423D-0-0] [lt=52][errcode=-4013] fail to wait block io, may be retry(macro_id_={[ver=1,mode=0,seq=6630924][2nd=4387]}, ret=-4013)

[2026-07-01 00:44:54.451951] WDIAG [STORAGE] get_loaded_block_data (ob_micro_block_handle_mgr.cpp:239) [2182941][T1002_L0_G0][T1002][YB427F000001-0006450246EE423D-0-0] [lt=12][errcode=-4013] Fail to wait micro block io(ret=-4013)

[2026-07-01 00:44:54.452164] INFO [COMMON] wash (ob_kvcache_store.cpp:529) [92][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash time detail, (compute_wash_size_time=745, refresh_score_time=81, wash_time=375, reclaim_time=333, reclaimed_size=22888448)

[2026-07-01 00:44:54.456000] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:706) [92][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Cache wash clean map node details(ret=0, clean_node_count=416, clean_time=0, clean_start_pos=251656, clean_num=62914)

[2026-07-01 00:44:54.456088] INFO [COMMON] wash (ob_kvcache_hazard_domain.h:326) [92][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] allocator wash time (wash_time_us=65, freed_blocks_num=0, memory_efficiency=9.777645100593428396e-01)

[2026-07-01 00:44:54.470269] INFO [PALF] submit_log (palf_handle_impl.cpp:446) [789][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT APPEND DATA SIZE](this={palf_id:1, self:“127.0.0.1:2882”, has_set_deleted:false}, append size=2107)

[2026-07-01 00:44:54.470309] INFO [LIB] stat (utility.h:1219) [789][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT APPEND COST TIME](cur_stat_count=5, stat_interval=1000000, avg (count/cost)=0, this=0xfffb643e35d0, extra_info=“id:1”)

[2026-07-01 00:44:54.470322] INFO [STORAGE.TRANS] check_all_readonly_tx_clean_up (ob_ls_tx_service.cpp:356) [789][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=10] wait_all_readonly_tx_cleaned_up cleaned up success(ls_id={id:1})

[2026-07-01 00:44:54.471073] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:372) [732][T1002_TransTime][T1002][YB427F000001-00064501E1FFC0B2-0-0] [lt=32] handle tx commit timeout(ret=0, tx_id={txid:178700330}, ref_cnt=4, cb_executed=false)

[2026-07-01 00:44:54.471118] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:372) [732][T1002_TransTime][T1002][YB427F000001-00064501E1FFC0B2-0-0] [lt=27] handle tx commit timeout(ret=0, tx_id={txid:178700291}, ref_cnt=4, cb_executed=false)

[2026-07-01 00:44:54.471147] INFO [STORAGE.TRANS] handle_tx_commit_timeout (ob_trans_service_v4.cpp:372) [732][T1002_TransTime][T1002][YB427F000001-00064501E1FFC0B2-0-0] [lt=15] handle tx commit timeout(ret=0, tx_id={txid:178700311}, ref_cnt=4, cb_executed=false)

[2026-07-01 00:44:54.473781] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:93) [733][T1002_TransTime][T1002][YB427F000001-0006450273CCFF7E-0-0] [lt=501][errcode=-4273] after wash from cache, still can’t alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273)

[2026-07-01 00:44:54.473779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:845) [731][T1002_DiskCB0][T1002][YB427F000001-00064501E1FFE3BB-0-0] [lt=283] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=6242304, size_to_wash=6291456)

AntTech_OXEPTD · 2026 年7 月 2 日 08:49

重启能搞定不。

oceanfish · 2026 年7 月 2 日 09:03

能看到是哪个语句引发的吗？可以把语句发出来看看

淇铭 · 2026 年7 月 2 日 09:52

grep ‘[OOPS]’ observer.log*
注意[有反斜杠
能在根据这个搜一下么如果过滤的信息有把整个文件都发一下

wkk2025 · 2026 年7 月 2 日 10:22

顶一下

技术小趴菜 · 2026 年7 月 2 日 10:41

longzh85 · 2026 年7 月 2 日 12:08

没碰到过类似问题，学习一下

淇铭 · 2026 年7 月 2 日 15:26

从信息查看 [errcode=-4013] alloc failed reason is that server memory has reached the upper limit

(server_hold: 8588558336, server_limit: 8589934592, alloc_size: 2097152)
看着是节点的内存使用到上线了看着是 test 租户 Unit 16.25 GB + sys 2 GB，合计 18.25 GB，远超 8 GB Observer
SHOW PARAMETERS LIKE ‘memory_limit’;

SHOW PARAMETERS LIKE ‘memory_limit_percentage’;

SHOW PARAMETERS LIKE ‘system_memory’;
参数信息查一下

SELECT tenant_id, svr_ip, hold, free

FROM GV$OB_TENANT_MEMORY

ORDER BY hold DESC;

SELECT tenant_id, ctx_name, mod_name, hold/1024/1024 AS hold_mb

FROM GV$OB_MEMORY

ORDER BY hold DESC LIMIT 30; 信息在查一下

耶一耶 · 2026 年7 月 2 日 15:27

厉害了

Sunshining · 2026 年7 月 2 日 16:22

资源到达上限，扩下资源就可以了吧

AntTech_RWNFTN · 2026 年7 月 3 日 11:56

test 租户 Unit 16.25 GB 这个值是怎么来的？我用的是docker部署的

淇铭 · 2026 年7 月 3 日 16:21

根据unit配置的信息查询出来的你查一下这个信息
SELECT * FROM oceanbase.DBA_OB_UNIT_CONFIGS;

wkk2025 · 2026 年7 月 3 日 17:09

顶一下