[2025-04-03 14:01:08.065891] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.065916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.066049] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.066081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.066185] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.066357] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.066393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.066472] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.066609] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.066707] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.066727] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.066830] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.066944] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.066971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.067072] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.067181] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.067315] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.067341] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.067458] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.067635] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.067668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.067750] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.067852] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.067877] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.068020] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.068157] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.068190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.068347] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.068488] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.068520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.068604] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.068821] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.068840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.068926] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069031] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.069158] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069264] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069353] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069383] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.069477] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069568] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.069680] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069764] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069865] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:08.069953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.070728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.070920] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.071003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.071177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.071305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.071425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.071618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.071751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.071939] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.072096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.072250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.072409] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.072444] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.072580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.072726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.073078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.073221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.073547] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.073702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.073867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.074055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.074365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.074519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=8] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.074698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.074842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.075012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.075187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.075334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.075499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.075660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.076308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.076652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.076797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.077052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.077152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.084863] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A24-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.084898] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A24-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=702, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.091382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=41] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.091965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092225] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092572] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.092896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=37] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093051] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093337] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093398] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.093903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.094051] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.094214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.094553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.095025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.095777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.095913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.096090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.096234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.096257] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.096291] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.096326] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.096342] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.096757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.096874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.097083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.097458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.097730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.097864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.098255] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.098551] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.098563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.098846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.099025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.099035] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE36E-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.099178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.099330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.100096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.100186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.100296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.100490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.107420] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A26-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.107462] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A26-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1206, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.109600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.109972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.110078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.110085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.110230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.110240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.110414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.110422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.112463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.112483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.112605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.112622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.112690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.112726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.112790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.112906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.113587] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.114198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.114390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.114543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.115023] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.115229] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.115651] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.115825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.116014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.116194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.116511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.116638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.116988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.117137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.117312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.117459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.117626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.117802] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.117990] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.118135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.118431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.118589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.118734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.118899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.128995] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A28-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.129022] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A28-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=436, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.139562] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A29-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.139623] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A29-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=500, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.142905] INFO [ARCHIVE] get_send_task_ (ob_archive_sender.cpp:364) [18637][T1004_ArcSender][T1004][YB42AC140005-000631D6C0DCB54C-0-0] [lt=23] task not exist, wait time(time_cost=100065) [2025-04-03 14:01:08.149339] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=32] ====== check clog disk timer task ====== [2025-04-03 14:01:08.149383] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=30] get_disk_usage(ret=0, capacity(MB):=9216, used(MB):=563) [2025-04-03 14:01:08.149554] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=18] cannot_recycle_log_size statistics(cannot_recycle_log_size=322182499, threshold=2899102924, need_update_checkpoint_scn=false) [2025-04-03 14:01:08.149985] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A2A-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.150011] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A2A-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=289, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.152081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152351] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152792] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.152969] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153383] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.153990] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=44] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.154974] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.155122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.155143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.155305] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4670569144, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:08.155348] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:08.155618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.155636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.155878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.155894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156089] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.156989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157005] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157195] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=42] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.157957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.158169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.158185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.158325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.158341] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.158576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.158591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.158848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.158865] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.159921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.160382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.161310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.162858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.163163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.163352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.163982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.164163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.164299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.164476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.164644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.164734] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=29] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.165296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.165501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.165878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.166100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.166290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.166471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.166592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A1F-0-0] [lt=115] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.166747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.167105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.167331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.168077] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.168320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.168332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.168680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.169008] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.169201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.169379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.169515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.169607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.169757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.170089] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.170106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.173440] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A20-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.173479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A20-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.174159] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A20-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.191698] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A2E-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.196321] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.196362] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.196377] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.196391] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.196434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.209608] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4803124137, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:08.213217] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.213337] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28176 [2025-04-03 14:01:08.215181] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.223323] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:08.223380] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=49] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28178 [2025-04-03 14:01:08.223458] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28178 [2025-04-03 14:01:08.223476] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.224895] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A31-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.224995] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A31-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=737, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.234953] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.235011] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.235288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.235516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.235740] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A32-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.235803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.235840] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A32-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=617, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.236518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.237116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.237391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.238017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.238505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.238845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.238902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.239413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.239589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.240079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.240335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.240573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.240751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.240956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.241106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.241296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.241501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.241638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.241948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.242942] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.243072] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.243202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.243177] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18193][T1002_TenantMet][T1002][Y0-0000000000000000-0-0] [lt=97] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:545, free_obj_hold_cnt:71, allocator used:2449216, allocator total:2545920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:59, free_obj_hold_cnt:5, allocator used:4197824, allocator total:4197824}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6743744, tablets_mem_limit=107374180, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:4, free_obj_hold_cnt:0, allocator used:7936, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=604) [2025-04-03 14:01:08.243315] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.243495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.243666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.243833] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.243968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.244080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.247709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.250359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.250617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.251019] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.251327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.251450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8C9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.252270] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1962) [17937][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=1] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-04-03 14:01:08.253389] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.253425] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28180 [2025-04-03 14:01:08.253524] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28180 [2025-04-03 14:01:08.253541] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.255312] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2449) [17937][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=1] Dumping All Cache Objs(alloc_obj_list.count()=67, alloc_obj_list=[{obj_id:3063, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175493}, {obj_id:3065, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176543}, {obj_id:3069, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186550}, {obj_id:3070, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182609}, {obj_id:3071, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209465}, {obj_id:3072, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182625}, {obj_id:3073, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187423}, {obj_id:3075, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191136}, {obj_id:3076, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182721}, {obj_id:3077, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183848}, {obj_id:3265, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191770}, {obj_id:3284, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199377}, {obj_id:3314, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197276}, {obj_id:3369, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186147}, {obj_id:3370, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184862}, {obj_id:3371, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187646}, {obj_id:3372, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:3373, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188001}, {obj_id:3374, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185069}, {obj_id:3375, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187994}, {obj_id:3376, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185011}, {obj_id:3377, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191261}, {obj_id:3379, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189071}, {obj_id:3380, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189114}, {obj_id:3381, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176578}, {obj_id:3382, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181961}, {obj_id:3388, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186886}, {obj_id:3389, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199789}, {obj_id:3390, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178193}, {obj_id:3391, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:436996}, {obj_id:3392, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:309921}, {obj_id:3393, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:251066}, {obj_id:3394, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176712}, {obj_id:3395, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175812}, {obj_id:3396, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:305539}, {obj_id:3397, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246674}, {obj_id:3398, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179377}, {obj_id:3399, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:227956}, {obj_id:3400, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187525}, {obj_id:3401, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196150}, {obj_id:3402, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173385}, {obj_id:3403, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:303583}, {obj_id:3404, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223881}, {obj_id:3405, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230686}, {obj_id:3406, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198295}, {obj_id:3407, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198293}, {obj_id:3408, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192308}, {obj_id:3409, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213222}, {obj_id:3410, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197338}, {obj_id:3411, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198558}, {obj_id:3412, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188271}, {obj_id:3413, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192389}, {obj_id:3414, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183025}, {obj_id:3416, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180892}, {obj_id:3417, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173048}, {obj_id:3418, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175597}, {obj_id:3420, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:4992223}, {obj_id:3421, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179536}, {obj_id:3422, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187623}, {obj_id:3423, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187597}, {obj_id:3424, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181588}, {obj_id:3425, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178170}, {obj_id:3426, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179454}, {obj_id:3427, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204521}, {obj_id:3428, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187680}, {obj_id:3429, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189293}, {obj_id:3430, tenant_id:1001, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198317}]) [2025-04-03 14:01:08.255487] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2702) [17937][T1001_PlanCache][T1001][Y0-0000000000000000-0-0] [lt=172] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:08.255460] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.255509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.255608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.257032] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4670670540, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:08.258170] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A34-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.258253] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A34-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=932, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.259957] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.260003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.260262] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.260455] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.260741] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.260957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.261221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.261452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.261869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.262061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.262079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.262206] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.266534] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.266580] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=39] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28182 [2025-04-03 14:01:08.266664] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28182 [2025-04-03 14:01:08.266709] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=41] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.267262] INFO [STORAGE] schedule_all_tablets_minor (ob_tenant_tablet_scheduler.cpp:561) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] start schedule all tablet minor merge(minor_ls_tablet_iter_={ls_idx:-1, ls_ids:[{id:1}], tablet_idx:0, tablet_ids_.count():0, schedule_tablet_cnt:0, max_batch_tablet_cnt:50000, report_scn_flag:false}) [2025-04-03 14:01:08.267503] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.267529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.267907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.267991] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.268084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.268142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.268229] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.268440] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.268533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.268608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.268678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.268752] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.270283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.274379] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=1] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=1) [2025-04-03 14:01:08.274753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.276195] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.278793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.282540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.283352] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.285469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.287586] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.292147] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A37-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=825, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.296418] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.296462] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.296484] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.296503] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.299473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.299639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.301624] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.302404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.311200] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4803225455, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:08.314455] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.314617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.316593] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.316900] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=2] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28192 [2025-04-03 14:01:08.332700] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A25-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.333180] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.333348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.337902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.340996] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.349968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.355446] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.356002] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.356185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.363031] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:08.363928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.363971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.367540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.367710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.370308] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.371990] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A3E-0-0] [lt=0][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.372659] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:08.373565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.381056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A29-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.384972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A29-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.392501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.396502] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.396546] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.396567] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.396615] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.402067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.405816] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A2B-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.417541] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A42-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.424208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.428155] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [18401][T1003_LockWaitM][T1003][Y0-0000000000000000-0-0] [lt=46] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.431295] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE77-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.431535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.433692] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE77-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.433979] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE77-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.435933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.436096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.438213] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28206 [2025-04-03 14:01:08.441795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18186][T1002_FreInfoRe][T1002][YB42AC140005-000631D6C3FCEE78-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.446355] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=2] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28210 [2025-04-03 14:01:08.448636] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.450589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.452763] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A45-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.452875] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A45-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2098, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:08.453084] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.454507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.456288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.456665] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4670870579, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:08.459212] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.471454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.474268] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.474534] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=47] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=0) [2025-04-03 14:01:08.484187] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA65E-0-0] [lt=740][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.484793] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Wash time detail, (compute_wash_size_time=213, refresh_score_time=110, wash_time=20) [2025-04-03 14:01:08.486095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.488389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.491959] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=80] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=7084, clean_start_pos=880796, clean_num=62914) [2025-04-03 14:01:08.493426] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=30][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:08.493465] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=37][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660075703884, now=[mts=1743660067284202], now0=[mts=1743660067284202], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:08.493496] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=27][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a4335d80, tx_id:{txid:76175005}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660067276988, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:08.493587] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=84][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73961321a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a4335d80}) [2025-04-03 14:01:08.493651] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=62][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a4335d80, tx_id:{txid:76175005}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660067276988, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:08.493702] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=48] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.493788] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=83][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:08.493830] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=39][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:08.493843] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:08.493854] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:08.493867] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:08.493906] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=36][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=17, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:08.494739] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28224 [2025-04-03 14:01:08.494951] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2DB-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.496602] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.496732] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=52][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.498002] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.498080] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=2][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.498119] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.499273] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.502216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.502704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A2C-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.503957] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2DB-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.504333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.505963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.506611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.509548] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.509835] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.511013] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:17, local_retry_times:17, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:08.511118] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=82][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:08.511143] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=23][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:08.511154] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:08.514265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.517733] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.517789] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:08.518255] INFO [ARCHIVE] get_send_task_ (ob_archive_sender.cpp:364) [18252][T1002_ArcSender][T1002][YB42AC140005-000631D6BE5CB54C-0-0] [lt=31] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.520453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.521559] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:08.526804] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.536575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.539671] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.543902] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3A4D-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:08.546345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.555640] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4670970167, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:08.556442] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.567188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.568979] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.573218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.575847] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.578763] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28236 [2025-04-03 14:01:08.590159] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.590963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.596670] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.596700] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.596715] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.596727] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.603012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.607421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A2E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.613923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.617017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.620111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A2F-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.624681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.632327] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.633630] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28244 [2025-04-03 14:01:08.636413] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17893][T1001_Occam][T1][YB42AC140005-000631D6BD7CDA1D-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.637865] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.641945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.643062] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=48][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:08.643110] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=46][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660067454635], now0=[mts=1743660067454635], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:08.643153] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=38][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719462}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660067444550, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:08.643240] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=80][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:08.643283] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=41][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719462}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660067444550, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:08.643410] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=123] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.643563] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=150][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:08.643591] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=25][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:08.643604] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=12][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:08.643618] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:08.643630] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:08.643651] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=11, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:08.643721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.644960] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18193][T1002_TenantMet][T1002][Y0-0000000000000000-0-0] [lt=78] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:545, free_obj_hold_cnt:71, allocator used:2449216, allocator total:2545920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:59, free_obj_hold_cnt:5, allocator used:4197824, allocator total:4197824}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=6743744, tablets_mem_limit=107374180, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:4, free_obj_hold_cnt:0, allocator used:7936, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=604) [2025-04-03 14:01:08.648444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.653167] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=2] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28252 [2025-04-03 14:01:08.654856] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:11, local_retry_times:11, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:08.654937] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=57][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:08.654955] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=17][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:08.654969] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=12][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:08.658051] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.659203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.662448] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.666124] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.666159] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:08.668891] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:08.676132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.678497] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:686) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=37] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.679385] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=2] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28262 [2025-04-03 14:01:08.682144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A30-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.685563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A31-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.692088] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.692429] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.693238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.693874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.696902] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.696956] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.696983] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.697003] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.703678] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.710172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A34-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.712275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.712736] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.714229] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.727100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.735715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745173] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745244] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745313] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745380] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745402] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745455] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745502] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.745830] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1001}, {id:1003}]) [2025-04-03 14:01:08.746118] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746125] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746128] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746137] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746150] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746153] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746174] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746177] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746198] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746200] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=2][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746221] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746223] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=2][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746244] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746246] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=2][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746343] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746347] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746368] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746371] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746377] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746390] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746394] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746399] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746422] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746425] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746447] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746448] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746453] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746415] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=81][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:55393, tid:18133}]) [2025-04-03 14:01:08.746453] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746475] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746495] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=20][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746520] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746531] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746543] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746548] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746565] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746574] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746598] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746604] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746613] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746619] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746627] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746633] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746656] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746664] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746678] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746695] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746695] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746700] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746723] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746731] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746751] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746754] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746759] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746759] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746789] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746799] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746832] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746834] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746839] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746842] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746864] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746879] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746903] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746911] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746934] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746942] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746963] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.746966] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.746970] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746973] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.746992] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747000] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747024] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747029] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747035] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747032] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747072] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747081] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747103] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747112] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747120] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747124] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747134] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747138] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747160] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747168] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747191] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747198] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747221] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747222] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747232] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747236] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747261] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747269] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747289] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747292] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747296] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747299] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747319] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747322] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747354] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=2][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747357] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747371] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747378] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747390] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747396] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747437] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747446] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747459] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747467] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747469] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747473] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747496] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747504] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747528] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747529] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747535] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747537] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747563] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747571] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747581] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747591] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747594] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747614] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=20][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.747646] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747646] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747650] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747653] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747674] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747680] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747699] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747702] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747707] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747709] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747732] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747741] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747766] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747766] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747772] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747775] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747806] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747822] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747837] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747845] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747846] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747849] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747880] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747888] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747902] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747909] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747914] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747911] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747937] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747946] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747972] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.747969] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.747981] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.747983] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748007] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748016] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748050] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748053] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.748057] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748059] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748080] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748089] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748113] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748121] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748120] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.748126] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748144] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748147] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748173] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.748178] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748183] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748183] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748206] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748214] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748238] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748246] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748270] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748278] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748287] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.748298] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748310] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748315] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748338] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748356] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748378] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748377] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.748387] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748390] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748428] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748438] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748443] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:08.748448] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2A35-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.748466] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:08.748471] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:08.753790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.764233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.764700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.772547] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.774635] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [18419][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-04-03 14:01:08.774679] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18419][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=34] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.781147] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.794992] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28282 [2025-04-03 14:01:08.796953] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.796978] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.796988] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.796997] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.797776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.801926] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1543-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:08.802155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.804923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.807203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.811995] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17513][MetaLocAsyncUp0][T1][YB42AC140005-000631D6BC0E7DA8-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.813590] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1543-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.816322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.819904] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.829186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.831973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.832856] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18500][T1004_LogUpdate][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:08.833420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.837166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.844168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18427][T1003_MergeLoop][T1003][YB42AC140005-000631D6C9ACB5AD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.844662] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.844824] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.844993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.845160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.845317] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.845830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.846134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.846525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.846662] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=41][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4401, tid:17484}]) [2025-04-03 14:01:08.847019] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.847169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.847288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.847428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.847945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.848084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.848435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.848494] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2A3B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:08.850701] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:08.850832] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:08.852308] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:08.854916] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.854972] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=47] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28288 [2025-04-03 14:01:08.855111] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=53] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28288 [2025-04-03 14:01:08.855137] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.855859] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.855879] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.855924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.855935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.855967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.855978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856008] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856173] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856366] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856424] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:08.856635] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:08.856651] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:08.856660] WDIAG [STORAGE.TRANS] post_cluster_heartbeat_rpc_ (ob_tenant_weak_read_service.cpp:911) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] post cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_service_master="172.20.0.3:2882", tenant_id_=1001, req={req_server:"172.20.0.5:2882", version:{val:1743659476426054000, v:0}, valid_part_count:0, total_part_count:1, generate_timestamp:1743660068855795}, cluster_service_tablet_id={id:226}) [2025-04-03 14:01:08.856685] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4013] tenant weak read service cluster heartbeat RPC fail(ret=-4013, rcode={code:-4013, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="172.20.0.5:2882", cluster_service_tablet_id={id:226}) [2025-04-03 14:01:08.856713] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] post cluster heartbeat rpc fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, local_server_version={val:1743659476426054000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1743660068855795) [2025-04-03 14:01:08.856722] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] tenant weak read service do cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1743660068656740, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="172.20.0.3:2882") [2025-04-03 14:01:08.856774] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=4671270747, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:08.866495] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.866542] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28290 [2025-04-03 14:01:08.866621] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28290 [2025-04-03 14:01:08.866649] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.873516] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.873558] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28292 [2025-04-03 14:01:08.873644] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28292 [2025-04-03 14:01:08.873658] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.878835] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.878841] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=1] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=9) [2025-04-03 14:01:08.878867] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28294 [2025-04-03 14:01:08.878922] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28294 [2025-04-03 14:01:08.878941] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.895667] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:08.895708] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28296 [2025-04-03 14:01:08.895845] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28296 [2025-04-03 14:01:08.895869] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:08.897108] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.897165] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.897188] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.897208] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.955899] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4671369696, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:08.955950] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:08.968322] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1001}]) [2025-04-03 14:01:08.997196] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.997245] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.997267] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:08.997287] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.047135] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=43][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:32035, tid:17513}]) [2025-04-03 14:01:09.056060] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.056149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=84] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056237] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.056291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=51] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056303] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056352] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.056368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056437] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.056484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056545] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.056599] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=52] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056782] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.056833] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.056966] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.057020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=51] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=58] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057251] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.057296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057444] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.057474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057619] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.057659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057732] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.057757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057867] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.057891] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.057957] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.058001] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058097] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.058121] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058215] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.058239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058311] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.058356] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058447] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.058466] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058764] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.058791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.058922] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4671472938, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.058959] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.078950] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=8) [2025-04-03 14:01:09.080308] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=1][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.080354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=41] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.080379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.080394] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=14][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.080700] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.080781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=77] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.080798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.080983] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.081055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=68] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.081071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.081298] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.081350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=49] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.081362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.081890] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.081952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=58] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.081964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.082194] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.082228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.082240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.082574] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.082610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.082623] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.082870] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.082919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.082929] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.083239] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.083287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=45] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.083297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.083518] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.083565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=45] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.083577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.083782] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.083846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=61] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.083880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.084127] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.084156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.084167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.084493] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.084574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=78] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.084590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.084741] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.084787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=44] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.084797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.084961] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.085011] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=46] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.085024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.085561] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=1][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.085618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=54] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.085634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.085882] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.085911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.085921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.086191] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.086224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.086236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.086487] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.086534] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=44] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.086544] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.086857] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=1][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.086908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=48] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.086922] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.087176] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.087211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.087220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.087390] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.087411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.087421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.087603] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.087624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.087631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.087796] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.087848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=50] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.087858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.088540] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.088594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=52] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.088605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.088833] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.088868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.088875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089057] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.089078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089241] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.089260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089568] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.089603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089837] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.089863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.089976] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=1][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.089998] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090026] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090329] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.090355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090464] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=1][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.090482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090600] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=1][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.090617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090732] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.090754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090917] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=1][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.090941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.090950] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.091067] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=1][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.091090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.091098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.091310] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=0] [CACHE] len: 2251 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=9.182378 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.836476 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=1519.069730 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646258 | score=138702.584177 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1748.532534 | [2025-04-03 14:01:09.091330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.091346] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.091430] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F4-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.097271] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=28] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660065092875, last_first_register_time_us=1743648246712872, register_time_us=1743660065092875, curr_time_us=1743660069097268) [2025-04-03 14:01:09.097369] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=42][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.097402] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097504] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.097543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097595] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.097614] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097653] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.097679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097721] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.097734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097778] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.097794] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097802] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097864] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.097888] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097895] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097941] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.097958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.097964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098006] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.098031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098100] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.098117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098154] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.098176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098212] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.098228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098284] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.098302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098338] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:09.098362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.098390] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.20.0.5:2882") [2025-04-03 14:01:09.098408] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.098430] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.098445] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.098460] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.114629] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE36F-0-0] [lt=8579][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.156124] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=53] get wrs ts(ls_id={id:1}, delta=4671569700, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.156196] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=64] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.156230] INFO [STORAGE] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:117) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] wead read timestamp is too old(timestamp={val:1743655397585394700, v:0}, gts_scn={val:1743659480926054981, v:0}, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1743655397585394700, v:0}}) [2025-04-03 14:01:09.169439] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.169501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=58] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.169522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.169539] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=16][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.169741] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.169787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=43] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.169802] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.169991] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.170014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170161] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.170192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170339] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.170380] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170542] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.170564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170633] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.170659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170785] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.170806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.170983] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.171006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=46] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171380] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.171415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171427] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171559] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.171580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171691] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:09.171721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.171885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172008] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172022] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.172988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173002] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173118] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173623] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.173951] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.174152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174303] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174774] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.174805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.175086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.175120] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.175514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.175532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.175714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.175729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.175786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.175798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.176015] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.176031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.176105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.176116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.176687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.176707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.176764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.176775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.177138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.177157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.177407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.177435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.177602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.177617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.177755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.177769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178087] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178663] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178962] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.178972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179255] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179557] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179879] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.179986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180072] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180189] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.180993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181262] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181586] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181705] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.181787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182433] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182671] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.182762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.183654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.183673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.183793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.183806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.183936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.183959] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184115] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184572] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.184961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185859] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185880] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [17813][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=1138] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-04-03 14:01:09.185948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.185957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186118] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186206] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186561] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.186930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187004] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187136] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=107] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187616] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.187996] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188066] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188102] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188181] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188225] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188315] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188427] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188432] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188539] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188693] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188725] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188750] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188810] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.188846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.189977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.190007] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.190012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.190149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.190157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.190237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.190252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.190357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.190364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.197347] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.197428] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=69][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.197447] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.197472] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.216577] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 427.00; ) [2025-04-03 14:01:09.216652] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:09.216844] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=50] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:09.256247] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.256292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=51] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256984] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.256999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257424] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257439] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257557] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.257734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258060] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.258660] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.258758] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4671672514, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.258788] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.267794] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=6] ====== tenant freeze timer task ====== [2025-04-03 14:01:09.267877] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1555) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=68] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=25165824, ctx.total_memstore_used_=58720256, ctx.total_memstore_hold_=58720256, memstore_freeze_trigger_limit_=42998980, tenant_id=1001) [2025-04-03 14:01:09.267919] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1564) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] [TenantFreezer] oldest frozen memtable(list="") [2025-04-03 14:01:09.279146] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=103] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=7) [2025-04-03 14:01:09.297503] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.297592] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=82][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.297616] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.297634] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.348098] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.348141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=47] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348664] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348761] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.348973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349094] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349693] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349772] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.349897] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB929-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.356740] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=54] get wrs ts(ls_id={id:1}, delta=4671771077, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.356802] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=53] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.397700] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.397748] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.397770] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.397791] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.410930] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=1] ====== traversal_flush timer task ====== [2025-04-03 14:01:09.411163] INFO [MDS] flush (mds_table_mgr.cpp:172) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=36] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b73c068a510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616808737813004, v:0}, clog_base_lsn:{lsn:4697333760}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743622739127262315, v:0}, all_id_meta:{id_meta:[{limited_id:1743655270710910399, latest_log_ts:{val:1743655254062460505, v:0}}, {limited_id:35000001, latest_log_ts:{val:1743639632198700904, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:450, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:09.411421] INFO [MDS] flush (mds_table_mgr.cpp:172) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=145] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b73b7ede510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1743655263788752080, v:0}, clog_base_lsn:{lsn:3690762240}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743648258823515129, v:0}, all_id_meta:{id_meta:[{limited_id:1740569710308384976, latest_log_ts:{val:1740569693675557106, v:0}}, {limited_id:32000001, latest_log_ts:{val:1740551265169597432, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:1292, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:09.411639] INFO [MDS] flush (mds_table_mgr.cpp:172) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=138] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b73b7f94510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1743655484668882629, v:0}, clog_base_lsn:{lsn:3556552704}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743648258592483359, v:0}, all_id_meta:{id_meta:[{limited_id:1743494462089433682, latest_log_ts:{val:1743494445468388405, v:0}}, {limited_id:33000001, latest_log_ts:{val:1743494442089433683, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:171, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:09.411743] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=99] succeed to traversal_flush(ret=0, ls_cnt=3) [2025-04-03 14:01:09.449083] INFO [STORAGE] runTimerTask (ob_tenant_tablet_scheduler.cpp:234) [18045][T1001_InfoPoolR][T1001][Y0-0000000000000000-0-0] [lt=33] InfoPoolResizeTask(cost_ts=51) [2025-04-03 14:01:09.456894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.456922] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.456986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.456995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457276] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457925] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.457996] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.458863] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4671873368, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.458895] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.479287] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=38] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=6) [2025-04-03 14:01:09.497849] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.497890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.497934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=42] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=49] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.498963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499048] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499115] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499433] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499510] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.499554] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:4316157182}, prev_lsn={lsn:4316157060}, fetch_start_log_id=29484365, last_slide_log_id=29484364, fetch_log_size=41943040, accepted_mode_pid=624, last_fetch_req_time=1743660069499553, last_fetch_end_lsn={lsn:4358100222}, last_fetch_max_log_id=29486412, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:29484365, max_lsn:4316157304, max_scn:{val:1743655475525511037, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:4316157304}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:4274214264}}, last_submit_lsn:{lsn:4316157182}, last_submit_end_lsn:{lsn:4316157304}, last_submit_log_id:29484365, last_submit_log_pid:624, max_flushed_lsn:{lsn:4316157182}, max_flushed_end_lsn:{lsn:4316157304}, max_flushed_log_pid:624, committed_end_lsn:{lsn:4316157182}, last_slide_log_id:29484364, last_slide_scn:{val:1743655475423544390, v:0}, last_slide_lsn:{lsn:4316157060}, last_slide_end_lsn:{lsn:4316157182}, last_slide_log_pid:624, last_slide_log_accum_checksum:1508656143, last_fetch_end_lsn:{lsn:4358100222}, last_fetch_max_log_id:29486412, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660069499553, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b736e5ba530}) [2025-04-03 14:01:09.499674] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=109][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.499708] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.499778] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.499998] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500026] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500242] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500363] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500827] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.500999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.501009] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.501106] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1001, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:3574142328}, prev_lsn={lsn:3574142206}, fetch_start_log_id=29361636, last_slide_log_id=29361635, fetch_log_size=41943040, accepted_mode_pid=637, last_fetch_req_time=1743660069501105, last_fetch_end_lsn={lsn:3616085368}, last_fetch_max_log_id=29363683, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1001, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:29361635, max_lsn:3574142328, max_scn:{val:1743655277355487643, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:3574142328}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:3532199898}}, last_submit_lsn:{lsn:3574142206}, last_submit_end_lsn:{lsn:3574142328}, last_submit_log_id:29361635, last_submit_log_pid:637, max_flushed_lsn:{lsn:3574142206}, max_flushed_end_lsn:{lsn:3574142328}, max_flushed_log_pid:637, committed_end_lsn:{lsn:3574142328}, last_slide_log_id:29361635, last_slide_scn:{val:1743655277355487643, v:0}, last_slide_lsn:{lsn:3574142206}, last_slide_end_lsn:{lsn:3574142328}, last_slide_log_pid:637, last_slide_log_accum_checksum:3441221005, last_fetch_end_lsn:{lsn:3616085368}, last_fetch_max_log_id:29363683, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660069501105, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7378b5e530}) [2025-04-03 14:01:09.501199] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=86][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.501220] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.551012] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.556930] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=4671970614, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.556975] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.578105] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.597863] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.597910] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.597928] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.597943] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.634122] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:09.656954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.656970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657075] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657255] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657437] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657561] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657910] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.657973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.658027] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.658044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.658106] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-04-03 14:01:09.658158] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=4672072280, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.658175] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.661391] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.661451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=56] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.661479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.661654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.661676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.661910] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.661934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662424] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662557] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662969] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.662988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.663190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.663217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.663614] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.663659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.663879] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.663930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=50] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.664317] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.664344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.664476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.664498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.664624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.664647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.664887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.664914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.665110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.665137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.665288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.665310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.665456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.665479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.665636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.665655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.666178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.666202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.670308] ERROR issue_dba_error (ob_log.cpp:1875) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4013, file="ob_mysql_protocol_processor.cpp", line_no=135, info="fail to alloc memory for OMPKHandshakeResponse") [2025-04-03 14:01:09.670347] EDIAG [RPC.OBMYSQL] decode_hsr_body (ob_mysql_protocol_processor.cpp:135) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4013] fail to alloc memory for OMPKHandshakeResponse(alloc_size=588, ret=-4013) BACKTRACE:0x125fd785 0x50b6335 0x51ad5ed 0x51ad0af 0x519a224 0x51acf53 0x1339477e 0x4d023dd 0x4cfef86 0x128cdbed 0x2b72da6c4ea5 0x2b72da9d7b0d [2025-04-03 14:01:09.671621] ERROR issue_dba_error (ob_log.cpp:1875) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4013, file="ob_mysql_protocol_processor.cpp", line_no=135, info="fail to alloc memory for OMPKHandshakeResponse") [2025-04-03 14:01:09.671643] EDIAG [RPC.OBMYSQL] decode_hsr_body (ob_mysql_protocol_processor.cpp:135) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4013] fail to alloc memory for OMPKHandshakeResponse(alloc_size=588, ret=-4013) BACKTRACE:0x125fd785 0x50b6335 0x51ad5ed 0x51ad0af 0x519a224 0x51acf53 0x1339477e 0x4d023dd 0x4d0147d 0x4cff429 0x128cdbed 0x2b72da6c4ea5 0x2b72da9d7b0d [2025-04-03 14:01:09.679422] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=5) [2025-04-03 14:01:09.697958] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.698064] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=99][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.698089] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] re_register_parent reason(palf_id=1001, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660065603107, last_first_register_time_us=1743648246815801, register_time_us=1743660065603107, curr_time_us=1743660069698087) [2025-04-03 14:01:09.698214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.698378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.698482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.698630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.698763] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.703161] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE79-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.704347] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.704403] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.704470] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=0] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:09.714875] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=31] ====== check clog disk timer task ====== [2025-04-03 14:01:09.714917] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=30] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=315) [2025-04-03 14:01:09.715375] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=15] cannot_recycle_log_size statistics(cannot_recycle_log_size=129898528, threshold=483183820, need_update_checkpoint_scn=false) [2025-04-03 14:01:09.718431] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=20315) [2025-04-03 14:01:09.718464] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=20341) [2025-04-03 14:01:09.718481] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1001, self="172.20.0.5:2882") [2025-04-03 14:01:09.718507] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.718536] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.746094] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1002}, {id:1001}, {id:1}]) [2025-04-03 14:01:09.748956] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=70][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:15646, tid:18427}]) [2025-04-03 14:01:09.753898] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=28][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:09.753963] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=62][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660075703884, now=[mts=1743660068518993], now0=[mts=1743660068518993], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:09.753992] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=26][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a4335d80, tx_id:{txid:76175006}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660068510538, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:09.754153] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=152][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73961321a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a4335d80}) [2025-04-03 14:01:09.754189] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=34][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a4335d80, tx_id:{txid:76175006}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660068510538, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:09.754263] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=60] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f8054008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660068517063, tx_desc={this:0x2b73a4335d80, tx_id:{txid:76175006}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660068510538, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660045773894, use_das=false, nested_level=0, session={this:0x2b73961321a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a4335d80}, plan=0x2b7338b48050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:09.754382] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=116][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:09.754405] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=20][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:09.754416] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:09.754427] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:09.754437] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:09.754453] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=18, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:09.754471] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15] will sleep(sleep_us=18000, remain_us=6019414, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=18, v.err_=-4283, timeout_timestamp=1743660075773884) [2025-04-03 14:01:09.757435] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=4672171191, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.757520] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=77] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.772603] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=41][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:18, local_retry_times:18, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:09.772685] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=60][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:09.772700] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:09.772710] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:09.772788] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.772847] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.772863] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.772966] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.772989] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.773153] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.773176] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=22][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.773487] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.773508] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=21][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.773725] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.773761] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=35][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.773872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.773906] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=33][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.773916] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.773974] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.773984] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.774181] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.774194] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.774333] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.774345] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.774480] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.774492] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.774617] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.774630] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.774745] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.774757] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.774889] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.774903] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.775073] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.775103] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=30][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.775238] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.775250] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.775298] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.775308] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.775445] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.775458] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.775586] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.775598] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.775715] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.775728] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.775873] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.775885] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.776002] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.776014] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.776149] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.776161] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.776276] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.776288] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.776514] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.776528] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.776630] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.776649] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.776774] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.776786] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.776927] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.776939] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777086] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777100] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777185] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777197] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777281] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777293] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777424] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777436] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777520] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777532] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777623] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777634] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777717] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777729] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777829] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777843] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.777943] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.777962] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778088] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778101] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778183] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778195] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778277] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778289] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778372] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778383] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778598] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778610] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778701] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778713] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778795] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778825] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=29][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778909] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778922] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.778982] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.778992] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779049] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779063] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779109] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779117] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779254] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779266] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779343] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779355] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779424] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779436] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779496] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779508] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779592] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779604] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779688] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779719] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=30][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779768] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779777] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779882] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779894] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.779983] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.779995] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.780110] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.780121] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.780169] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.780178] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.780293] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.780305] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.780388] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:09.780400] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:09.780449] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.780479] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:09.782793] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.788166] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:09.798133] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.798181] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.798203] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.798243] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.803476] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-04-03 14:01:09.803528] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=29] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-04-03 14:01:09.803557] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=25] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-04-03 14:01:09.803567] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=9] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-04-03 14:01:09.803577] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=8] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-04-03 14:01:09.803587] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=7] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=2367) [2025-04-03 14:01:09.806212] ERROR issue_dba_error (ob_log.cpp:1875) [17849][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4013, file="ob_multi_version_garbage_collector.cpp", line_no=222, info="repeat refresh failed too much time") [2025-04-03 14:01:09.806247] EDIAG [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:222) [17849][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4013] repeat refresh failed too much time(ret=-4013, this={this:0x2b72e57b8b30, last_study_timestamp:1743654908659042, last_refresh_timestamp:1743655208697194, last_reclaim_timestamp:1743659769442230, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:true, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1743654782789085444, v:0}, is_inited:true}, current_timestamp=1743660069580902) BACKTRACE:0x125fd785 0x50b6335 0x51fab28 0x51fa63b 0x51fa57e 0x51fa3a6 0xef83d9e 0xef927e9 0x5209517 0x5206c75 0x520639e 0x7662b60 0x128d17e1 0x128cdbbf 0x2b72da6c4ea5 0x2b72da9d7b0d [2025-04-03 14:01:09.832837] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.839534] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.841241] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.845314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.846064] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.846695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.846922] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.848108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.848458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.849133] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:1979, tid:17463}]) [2025-04-03 14:01:09.849298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.849472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.850350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.850537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.850690] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:09.850852] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:09.851159] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.851522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.851697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.851889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.852076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.852227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.852378] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=27] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.852700] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:09.852717] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=15][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:09.852729] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1544-0-0] [lt=11][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4013, ret=-4013) [2025-04-03 14:01:09.852740] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2025-04-03 14:01:09.852748] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:09.852754] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:09.852760] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] failed to read without check sys variable(ret=-4013, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1001, check_sys_variable=false, snapshot_timestamp=-1) [2025-04-03 14:01:09.852771] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2025-04-03 14:01:09.852854] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] load failed(ret=-4013, for_update=false) [2025-04-03 14:01:09.852868] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] core_table load failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:09.852875] WDIAG [SHARE] get_core_schema_version (ob_global_stat_proxy.cpp:390) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] get failed(ret=-4013) [2025-04-03 14:01:09.852881] WDIAG [SHARE.SCHEMA] get_core_version (ob_schema_service_sql_impl.cpp:857) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] get_core_schema_version failed(ret=-4013) [2025-04-03 14:01:09.852890] WDIAG [SHARE.SCHEMA] can_read_schema_version (ob_schema_service_sql_impl.cpp:6676) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] failed to get core schema version(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", schema_status={tenant_id:1001, snapshot_timestamp:-1, readable_schema_version:-1}) [2025-04-03 14:01:09.852917] WDIAG [SHARE.SCHEMA] fetch_schema (ob_schema_cache.cpp:939) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] incremant schema is not readable now, waiting and retry(ret=-4013, retry_times=0, schema_version=1) [2025-04-03 14:01:09.852925] WDIAG [SHARE.SCHEMA] get_schema (ob_multi_version_schema_service.cpp:636) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] fetch schema failed(tenant_id=1001, schema_type=15, schema_id=1001, schema_version=1, ret=-4013) [2025-04-03 14:01:09.852937] WDIAG [SHARE.SCHEMA] get_schema (ob_schema_getter_guard.cpp:4562) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] get schema failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", schema_status={tenant_id:1001, snapshot_timestamp:-1, readable_schema_version:-1}, schema_type=15, schema_id=1001, schema_version=1, specified_version=-1) [2025-04-03 14:01:09.852952] WDIAG [SHARE.SCHEMA] get_sys_variable_schema (ob_schema_getter_guard.cpp:434) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] fail to get sys var schema(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001) [2025-04-03 14:01:09.852959] WDIAG [SQL.SESSION] get_global_sys_variable (ob_basic_session_info.cpp:1003) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] get sys variable schema failed(ret=-4013) [2025-04-03 14:01:09.852966] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:281) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] failed to get global sys variable(ret=-4013, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-04-03 14:01:09.853000] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:74) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4013] failed to get mem limit(ret=-4013, tenant_id=1001, mem_limit=53687091, config_mem_limit_=16777216) [2025-04-03 14:01:09.857404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.857900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.858006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.858176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.858222] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:09.858232] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:09.858238] WDIAG [STORAGE.TRANS] post_cluster_heartbeat_rpc_ (ob_tenant_weak_read_service.cpp:911) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4][errcode=-4013] post cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_service_master="172.20.0.3:2882", tenant_id_=1001, req={req_server:"172.20.0.5:2882", version:{val:1743659476426054000, v:0}, valid_part_count:0, total_part_count:1, generate_timestamp:1743660069857370}, cluster_service_tablet_id={id:226}) [2025-04-03 14:01:09.858254] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] tenant weak read service cluster heartbeat RPC fail(ret=-4013, rcode={code:-4013, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="172.20.0.5:2882", cluster_service_tablet_id={id:226}) [2025-04-03 14:01:09.858272] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] post cluster heartbeat rpc fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, local_server_version={val:1743659476426054000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1743660069857370) [2025-04-03 14:01:09.858283] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] tenant weak read service do cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1743660069658098, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="172.20.0.3:2882") [2025-04-03 14:01:09.879533] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=4) [2025-04-03 14:01:09.898233] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.898284] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.898304] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.898322] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.906029] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=35][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:09.906062] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=32][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660068666467], now0=[mts=1743660068666467], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:09.906083] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719463}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660068653982, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:09.906122] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=36][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:09.906143] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719463}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660068653982, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:09.906163] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f7dce008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660068628556, tx_desc={this:0x2b7399219100, tx_id:{txid:47719463}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660068653982, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660054039470, use_das=false, nested_level=0, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}, plan=0x2b73976ba050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:09.906215] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=51][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:09.906260] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=43][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:09.906267] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=6][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:09.906271] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=3][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:09.906276] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=4][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:09.906282] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=5][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=12, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:09.906295] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10] will sleep(sleep_us=12000, remain_us=14133166, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=12, v.err_=-4283, timeout_timestamp=1743660084039460) [2025-04-03 14:01:09.918365] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:12, local_retry_times:12, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:09.918404] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=23][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:09.918410] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=5][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:09.918415] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=4][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:09.918435] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=4][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:09.918586] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:09.918610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918651] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918824] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.918877] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924542] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924727] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:09.924840] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.924857] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=16][errcode=-4013] oops, alloc failed, tenant_id=1, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=9223372036854775807, tenant_hold=606806016, tenant_limit=2147483648 [2025-04-03 14:01:09.924870] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=-4013] allocate memory fail(attr=tenant_id=1, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:09.924884] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=13][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:09.924893] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660069918552], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:09.924907] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=13][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660069918552], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:09.924917] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=9][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:09.944157] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1719][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:09.957509] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4672372027, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:09.957551] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:09.968454] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1}]) [2025-04-03 14:01:09.998312] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.998355] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.998373] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:09.998392] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11] [PALF_DUMP](palf_id=1003, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1003, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:615432, max_lsn:74920351, max_scn:{val:1743655479584546174, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:74920351}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:74671177}}, last_submit_lsn:{lsn:74920229}, last_submit_end_lsn:{lsn:74920351}, last_submit_log_id:615432, last_submit_log_pid:14, max_flushed_lsn:{lsn:74920229}, max_flushed_end_lsn:{lsn:74920351}, max_flushed_log_pid:14, committed_end_lsn:{lsn:74920229}, last_slide_log_id:615431, last_slide_scn:{val:1743655479483236415, v:0}, last_slide_lsn:{lsn:74920107}, last_slide_end_lsn:{lsn:74920229}, last_slide_log_pid:14, last_slide_log_accum_checksum:2102179633, last_fetch_end_lsn:{lsn:116863269}, last_fetch_max_log_id:617479, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660068499267, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7372ff0530}, [StateMgr]={this:0x2b7372ff4a30, self:"172.20.0.5:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:17}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:74671289}, scan_disk_log_finished:true, last_check_start_id:615432, is_changing_config_with_arb:false, reconfirm_start_time_us:1743648262361526, palf_role_change_cb:0x2b7372ffb2f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:14, prev:{version:1, config:{config_version:{proposal_id:13, config_seq:32}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:25, lock_time:1743614587856765}}, curr:{version:1, config:{config_version:{proposal_id:14, config_seq:33}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:25, lock_time:1743614587856765}}, prev_log_proposal_id:13, prev_lsn:{lsn:74671055}, prev_mode_pid:14}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:12, prev_lsn:{lsn:67040656}, prev_end_lsn:{lsn:67040778}, prev_mode_pid:13}, reconfig_barrier:{prev_log_proposal_id:12, prev_lsn:{lsn:67040656}, prev_end_lsn:{lsn:67040778}, prev_mode_pid:13}, persistent_config_version:{proposal_id:14, config_seq:33}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660066694760, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660066694760, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b7372ff1c70}, [ModeMgr]={palf_id:1003, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:14, mode_version:0, access_mode:1, ref_scn:{val:1743590085102110281, v:0}}, accepted_mode_meta:{version:1, proposal_id:14, mode_version:0, access_mode:1, ref_scn:{val:1743590085102110281, v:0}}, last_submit_mode_meta:{version:1, proposal_id:14, mode_version:0, access_mode:1, ref_scn:{val:1743590085102110281, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:17}, log_config_meta:{version:3, proposal_id:14, prev:{version:1, config:{config_version:{proposal_id:13, config_seq:32}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:25, lock_time:1743614587856765}}, curr:{version:1, config:{config_version:{proposal_id:14, config_seq:33}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:25, lock_time:1743614587856765}}, prev_log_proposal_id:13, prev_lsn:{lsn:74671055}, prev_mode_pid:14}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:0, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:14, mode_version:0, access_mode:1, ref_scn:{val:1743590085102110281, v:0}}}, log_meta_storage:{log_tail:{lsn:49152}, readable_log_tail:{lsn:49152}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1003/meta", dir_fd:125, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67055616, block_header_serialize_buf_:0x2b7372ff6bb8, flashback_version:0}, log_storage:{log_tail:{lsn:74920351}, readable_log_tail:{lsn:74920351}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1743648327251630861, v:0}, curr_block_id:1, palf_id:1003}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1003/log", dir_fd:126, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:59289185, block_header_serialize_buf_:0x2b7372ff8838, flashback_version:0}, palf_epoch:8, last_purge_throttling_ts:1743648262361055, this:0x2b7372ff5170}, [Reconfirm]={palf_id:1003, self:"172.20.0.5:2882", state:"FINISHED", new_proposal_id:13, prepare_log_ack_list:2"172.20.0.4:2882";"172.20.0.3:2882";, curr_paxos_follower_list:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.20.0.5:2882", majority_max_accept_pid:12, majority_max_lsn:{lsn:67040778}, saved_end_lsn:{lsn:67040778}, last_submit_prepare_req_time_us:1743648262362091, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b7372ff4bb0}) [2025-04-03 14:01:09.998894] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=492] [PALF_DUMP](palf_id=1003, self="172.20.0.5:2882", ack_info_list=[{member:{server:"172.20.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671055}}, {member:{server:"172.20.0.4:2882", timestamp:1, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671055}}, {member:{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671289}}]) [2025-04-03 14:01:09.998940] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4308) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=41] current log_task status(palf_id=1003, self="172.20.0.5:2882", log_id=615432, log_task={header:{begin_lsn:{lsn:74920229}, end_lsn:{lsn:74920351}, log_id:615432, min_scn:{val:1743655479584546174, v:0}, max_scn:{val:1743655479584546174, v:0}, data_len:66, proposal_id:14, prev_lsn:{lsn:74920107}, prev_proposal_id:14, committed_end_lsn:{lsn:74920229}, data_checksum:940009244, accum_checksum:2166578877, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1743659145927011, freeze_ts:1743659145927011, submit_ts:1743659145927013, flushed_ts:1743659145932829, gen_to_freeze cost time:0, gen_to_submit cost time:2, submit_to_flush cost time:5816}, ack_info_list=[{member:{server:"172.20.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671055}}, {member:{server:"172.20.0.4:2882", timestamp:1, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671055}}, {member:{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671289}}]) [2025-04-03 14:01:09.998994] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.023341] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=10132][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.049559] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:3081, tid:17422}]) [2025-04-03 14:01:10.057744] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.057796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.057837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058003] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.058025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058065] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058194] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.058214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058356] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.058382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058529] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.058549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058693] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.058728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058897] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.058925] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.058936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059077] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.059097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059257] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.059276] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059417] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.059449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059583] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.059602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059745] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.059764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059774] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059954] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.059982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.059992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.060058] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.060074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.060083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.060434] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.060471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.060485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.060845] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.060877] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.060889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061023] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.061061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061072] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061224] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.061265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061276] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061452] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.061478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061559] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.061575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061750] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.061778] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061942] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.061978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.061988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.062333] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.062359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.062371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.062779] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.062828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.062841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063017] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.063079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=59] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063300] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.063320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063419] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.063445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063549] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.063569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063587] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063698] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.063717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063848] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.063875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.063980] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:10.063999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.064032] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.064164] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4672478502, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.064199] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.071277] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:10.071324] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28298 [2025-04-03 14:01:10.071407] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28298 [2025-04-03 14:01:10.071436] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:10.079634] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=15] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=3) [2025-04-03 14:01:10.093532] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.093591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=54] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.093609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.093626] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=16][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:10.093858] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.093883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.093904] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094062] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.094084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094094] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094214] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.094240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094387] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.094413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094558] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.094584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094726] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.094753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094919] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.094939] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.094949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095114] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.095134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095275] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.095304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095435] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.095462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095598] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.095624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095772] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.095793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095826] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095958] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.095978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.095987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096119] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.096145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096283] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.096310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096448] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.096475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096655] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.096683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096835] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.096855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.096865] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097126] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.097150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097378] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.097406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097542] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.097568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097747] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.097773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097782] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097929] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.097948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.097965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.098508] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.098534] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.098546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.098691] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.098720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.098730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.098873] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.098900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.098912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.099069] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.099095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.099105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.099276] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.099304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.099315] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.099440] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.099460] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.099469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.102178] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.102230] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.102268] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.102285] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.139856] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [17821][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=1329] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b7309dfc2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-04-03 14:01:10.148218] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=1][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.148261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.148282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.148478] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.148530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=49] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.148544] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.148844] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.148887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.148901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.149089] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.149127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.149139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.149276] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.149302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.149323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.149493] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.149517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.149529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.149688] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=32] ====== check clog disk timer task ====== [2025-04-03 14:01:10.149748] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=46] get_disk_usage(ret=0, capacity(MB):=9216, used(MB):=563) [2025-04-03 14:01:10.149793] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.149839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=43] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.149850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.150127] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=0] [CACHE] len: 2250 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=5.422103 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=1.084421 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2528 | get_cnt= 9313 | score=896.995485 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646285 | score=105884.446931 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1909 | score=1032.490976 | [2025-04-03 14:01:10.150177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.150188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.150230] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F5-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.150420] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=19] cannot_recycle_log_size statistics(cannot_recycle_log_size=322182499, threshold=2899102924, need_update_checkpoint_scn=false) [2025-04-03 14:01:10.158552] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=4672572269, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.158607] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.179846] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE370-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.202221] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.202255] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.202267] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.202278] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.218409] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 705.00; ) [2025-04-03 14:01:10.218503] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=63] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:10.218689] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:10.258796] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=52][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:10.258873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=72] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.258909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259561] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=41] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259661] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.259994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260015] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.260704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.279763] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=2) [2025-04-03 14:01:10.281906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.281955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.282141] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.282212] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=23481) [2025-04-03 14:01:10.282236] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=23503) [2025-04-03 14:01:10.282334] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4672696342, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.282374] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.282441] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=-4389] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=23733, time_dist: do_cluster_heartbeat=23545) [2025-04-03 14:01:10.301744] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-04-03 14:01:10.301777] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=25] get_disk_usage(ret=0, capacity(MB):=1024, used(MB):=255) [2025-04-03 14:01:10.301821] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=11] cannot_recycle_log_size statistics(cannot_recycle_log_size=200526433, threshold=322122547, need_update_checkpoint_scn=false) [2025-04-03 14:01:10.302296] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.302326] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.302341] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.302354] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.358883] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=4672773128, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.358936] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.382478] INFO [STORAGE.TRANS] statistics (ob_location_adapter.cpp:72) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=6] location adapter statistics(renew_access=2626, total_access=2725, error_count=0, renew_rate=9.633162022e-01) [2025-04-03 14:01:10.402356] INFO [PALF] log_loop_ (log_loop_thread.cpp:175) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20] LogLoopThread round_cost_time(us)(round_cost_time=106) [2025-04-03 14:01:10.402385] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.402400] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.402409] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.402419] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.448103] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:10.448172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=64] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.448193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.448652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.448697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.449124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.449144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.449518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.449536] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.458920] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.458946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.459130] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.459146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.459231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.459244] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.459365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.459379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.459523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.459573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=49] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.479902] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=1) [2025-04-03 14:01:10.502443] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.502583] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=48][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:10.502636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=49] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.502685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.502862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.502887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.503016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.503053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.503183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.503207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.503346] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.503369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.503479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.503498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.515555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.515601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.515921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.515938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516147] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.516571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.527648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.527683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.527762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.527776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.527869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.527906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528351] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528398] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528576] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.528606] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=26102) [2025-04-03 14:01:10.528619] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=26113) [2025-04-03 14:01:10.528633] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1002, self="172.20.0.5:2882", dest="172.20.0.4:2882", fetch_log_count=2048, fetch_start_lsn={lsn:74011080}, prev_lsn={lsn:74010958}, fetch_start_log_id=607966, last_slide_log_id=607965, fetch_log_size=41943040, accepted_mode_pid=1, last_fetch_req_time=1743660070528631, last_fetch_end_lsn={lsn:115954120}, last_fetch_max_log_id=610013, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1002, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:608085, max_lsn:74025692, max_scn:{val:1743655277950177453, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:74011080}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:32082652}}, last_submit_lsn:{lsn:74010958}, last_submit_end_lsn:{lsn:74011080}, last_submit_log_id:607965, last_submit_log_pid:1, max_flushed_lsn:{lsn:74010958}, max_flushed_end_lsn:{lsn:74011080}, max_flushed_log_pid:1, committed_end_lsn:{lsn:74011080}, last_slide_log_id:607965, last_slide_scn:{val:1743655265068231480, v:0}, last_slide_lsn:{lsn:74010958}, last_slide_end_lsn:{lsn:74011080}, last_slide_log_pid:1, last_slide_log_accum_checksum:1277512058, last_fetch_end_lsn:{lsn:115954120}, last_fetch_max_log_id:610013, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660070528631, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b73ab3c0530}) [2025-04-03 14:01:10.528729] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=86][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.528759] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.528882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.528892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529254] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529442] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.529975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530015] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530154] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530346] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.530503] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1003, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:74920229}, prev_lsn={lsn:74920107}, fetch_start_log_id=615432, last_slide_log_id=615431, fetch_log_size=41943040, accepted_mode_pid=14, last_fetch_req_time=1743660070530502, last_fetch_end_lsn={lsn:116863269}, last_fetch_max_log_id=617479, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1003, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:615432, max_lsn:74920351, max_scn:{val:1743655479584546174, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:74920351}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:74671177}}, last_submit_lsn:{lsn:74920229}, last_submit_end_lsn:{lsn:74920351}, last_submit_log_id:615432, last_submit_log_pid:14, max_flushed_lsn:{lsn:74920229}, max_flushed_end_lsn:{lsn:74920351}, max_flushed_log_pid:14, committed_end_lsn:{lsn:74920229}, last_slide_log_id:615431, last_slide_scn:{val:1743655479483236415, v:0}, last_slide_lsn:{lsn:74920107}, last_slide_end_lsn:{lsn:74920229}, last_slide_log_pid:14, last_slide_log_accum_checksum:2102179633, last_fetch_end_lsn:{lsn:116863269}, last_fetch_max_log_id:617479, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660070530502, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7372ff0530}) [2025-04-03 14:01:10.530594] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=83][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.561983] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:103) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] tx gc loop thread is running(MTL_ID()=1) [2025-04-03 14:01:10.562200] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=37] update calc upper info once(calc_upper_info_={min_start_scn_in_ctx_:{val:1743655261104510952, v:0}, keep_alive_scn_:{val:1743655261104510952, v:0}, update_ts_:1743660070562196}) [2025-04-03 14:01:10.562258] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=43] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-04-03 14:01:10.562296] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info(lsn={lsn:13483027323}, scn={val:1743655265084940963, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1743655265088115, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12746101, ret_code_:0}}, next_to_submit_lsn_:{lsn:13483027323}, next_to_submit_scn_:{val:1743655265084940963, v:0}, base_lsn_:{lsn:13286744064}, base_scn_:{val:1743616801173178850, v:0}, iterator_:{iterator_impl:{buf_:0x2b738c205000, next_round_pread_size:2121728, curr_read_pos:537, curr_read_buf_start_pos:0, curr_read_buf_end_pos:537, log_storage_:{IteratorStorage:{start_lsn:{lsn:13483026786}, end_lsn:{lsn:13483027323}, read_buf:{buf_len_:2125824, buf_:0x2b738c205000}, block_size:67104768, log_storage_:0x2b7398e6fbf0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1743655265084940962, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:449, scn_:{val:1743655265084940962, v:0}, data_checksum:2328545553, flag:1}}, init_mode_version:0, accumulate_checksum:1426087181, curr_entry_is_padding:0, padding_entry_size:481, padding_entry_scn:{val:1743655265084940962, v:0}}}}}) [2025-04-03 14:01:10.564539] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=116] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616801173178850, v:0}, clog_base_lsn:{lsn:13286744064}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1743618797082332550, v:0}, all_id_meta:{id_meta:[{limited_id:1743655279117594735, latest_log_ts:{val:1743655262469466896, v:0}}, {limited_id:49000001, latest_log_ts:{val:1743605935907428857, v:0}}, {limited_id:360000001, latest_log_ts:{val:1743640357777174079, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:159, palf_env_:0x2b730a3fc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:7936, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:10.576967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.577003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.577074] WDIAG [LIB] ~Guard (ob_malloc_time_monitor.cpp:34) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] [MALLOC_TIME_MONITOR](cost_time=118214, time_guard=time dist: ALLOC_CHUNK_END=5, WASH_KVCACHE_END=101, ALLOC_CHUNK_END=124, WASH_KVCACHE_END=72, ALLOC_CHUNK_END=36, WASH_KVCACHE_END=61, ALLOC_CHUNK_END=69, WASH_KVCACHE_END=67, ALLOC_CHUNK_END=91, WASH_KVCACHE_END=102, ALLOC_CHUNK_END=55900, WASH_KVCACHE_END=131, SYNC_WASH_END=256, ALLOC_CHUNK_END=2, WASH_KVCACHE_END=78, ALLOC_CHUNK_END=154, attr=tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256) [2025-04-03 14:01:10.577152] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=118304) [2025-04-03 14:01:10.577170] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=118321) [2025-04-03 14:01:10.577203] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, do_cluster_heartbeat' cost too much time, used=118371, time_dist: get_server_version=2, post_cluster_heartbeat_rpc=118361) [2025-04-03 14:01:10.577264] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=4672991275, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.577291] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=3, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.577359] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-4389] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1001, thread_timer_task' cost too much time, used=118531, time_dist: do_cluster_heartbeat=118382) [2025-04-03 14:01:10.577385] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=4672991275, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.577403] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.579146] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.579724] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.602529] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.602605] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=70][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.602629] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.602664] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.635197] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=50] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:10.662080] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:110) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=103] try gc retain ctx [2025-04-03 14:01:10.677560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.677590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.677761] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.677779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.677864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.677878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678027] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678197] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678383] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678483] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [17830][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=1798] ====== traversal_flush timer task ====== [2025-04-03 14:01:10.678495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678508] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678786] INFO [MDS] flush (mds_table_mgr.cpp:172) [17830][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=31] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b7398e04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616801173178850, v:0}, clog_base_lsn:{lsn:13286744064}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1743618797082332550, v:0}, all_id_meta:{id_meta:[{limited_id:1743655279117594735, latest_log_ts:{val:1743655262469466896, v:0}}, {limited_id:49000001, latest_log_ts:{val:1743605935907428857, v:0}}, {limited_id:360000001, latest_log_ts:{val:1743640357777174079, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:159, palf_env_:0x2b730a3fc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:7936, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:10.678949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.678964] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [17830][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=169] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-04-03 14:01:10.679128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.679858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.680068] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=41] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=0) [2025-04-03 14:01:10.680214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.680254] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.680411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.680426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.680527] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-04-03 14:01:10.680601] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1}, delta=4673094923, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.680629] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.702629] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.702669] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.702692] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.702707] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.704663] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=46] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:10.746537] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=50] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1002}, {id:1}, {id:1001}]) [2025-04-03 14:01:10.751393] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:4584, tid:18605}]) [2025-04-03 14:01:10.760834] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [17546][Blacklist][T0][Y0-0000000000000000-0-0] [lt=4960] blacklist_loop exec finished(cost_time=66, is_enabled=true, send_cnt=0) [2025-04-03 14:01:10.777606] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=4673191937, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.777651] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.802785] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.802842] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.802863] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.802885] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] re_register_parent reason(palf_id=1003, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660066694760, last_first_register_time_us=1743648247119503, register_time_us=1743660066694760, curr_time_us=1743660070802883) [2025-04-03 14:01:10.802971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803018] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=45] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803082] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=62][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.803103] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.803190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803235] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.803245] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.803481] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803515] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.803523] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.803668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803701] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.803709] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.803844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.803876] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.803884] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.803982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.804000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.804010] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.804025] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.804187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.804208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.804221] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.804230] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.804339] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.804363] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.804374] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.804382] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.804770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.804795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.804827] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.804874] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=46][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.804978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.805000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.805013] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.805022] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.805158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.805177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.805188] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.805196] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.805358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.805379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.805411] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:10.805418] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.805470] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="172.20.0.5:2882") [2025-04-03 14:01:10.805493] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.850183] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:10.850949] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=9] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:10.851686] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=86][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:885, tid:17534}]) [2025-04-03 14:01:10.852137] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=11] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:10.861725] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:10.861765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=36] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.861782] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.861795] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.861805] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.862071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.862090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.862115] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=24][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.862141] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=25][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.862390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.862422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.862434] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.862443] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.862731] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.862751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.862763] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.862770] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.863127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.863150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.863163] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.863172] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.863603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.863622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.863632] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.863639] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.863966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.863983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.863994] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.864011] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.864494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.864510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.864521] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.864530] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.864896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.864911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.864923] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.864931] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.865334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.865350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.865370] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=18][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.865380] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.865643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.865655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.865666] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.865674] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.866053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.866066] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.866077] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.866087] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.866917] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.866942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.866953] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.866962] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.867684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.867700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.867732] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=31][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.867741] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.868399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.868429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.868441] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.868451] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.868914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.868928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.868939] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.868947] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.869356] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.869370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.869380] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.869389] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.870311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.870322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.870329] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.870343] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.870672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.870687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.870694] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.870700] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.870999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.871016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.871024] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.871043] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.871325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.871336] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.871349] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=13][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.871356] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.871624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.871635] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.871643] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.871650] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.871831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.871841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.871848] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:10.871853] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:10.872091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.872108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.872299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.872308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.872573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.872583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.872890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.872900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.873991] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.874168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.874177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.874489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.874499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.874711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.874720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.874760] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.874775] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=14][errcode=-4013] oops, alloc failed, tenant_id=1001, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=401567744, tenant_limit=536870912 [2025-04-03 14:01:10.874788] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:10.874796] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4013] rpc memory pool alloc memory failed(sz=1330, alloc_sz=16256) [2025-04-03 14:01:10.874805] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=-4013] alloc buffer fail(payload_sz=1146) [2025-04-03 14:01:10.874828] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=21][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:10.874843] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=1983166, timeout_timestamp=1743660072853127) [2025-04-03 14:01:10.874856] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=11][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:10.874866] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:10.874875] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:10.874883] WDIAG [SQL] open (ob_result_set.cpp:158) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:10.874899] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=12][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:10.874908] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2025-04-03 14:01:10.874920] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=11][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:10.874933] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:10.874967] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=21][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:10.874973] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:10.874980] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:10.875267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.875278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.875515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.875532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.875828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.875838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.876127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.876137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.876447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.876457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.876839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.876850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877011] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877708] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877877] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.877969] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=85] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.878950] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879027] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.879999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880081] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:10.880106] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:10.880162] WDIAG [STORAGE.TRANS] post_cluster_heartbeat_rpc_ (ob_tenant_weak_read_service.cpp:911) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=-4013] post cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_service_master="172.20.0.3:2882", tenant_id_=1001, req={req_server:"172.20.0.5:2882", version:{val:1743659476426054000, v:0}, valid_part_count:0, total_part_count:1, generate_timestamp:1743660070877644}, cluster_service_tablet_id={id:226}) [2025-04-03 14:01:10.880199] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-4013] tenant weak read service cluster heartbeat RPC fail(ret=-4013, rcode={code:-4013, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="172.20.0.5:2882", cluster_service_tablet_id={id:226}) [2025-04-03 14:01:10.880224] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] post cluster heartbeat rpc fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, local_server_version={val:1743659476426054000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1743660070877644) [2025-04-03 14:01:10.880244] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] tenant weak read service do cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1743660070680515, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="172.20.0.3:2882") [2025-04-03 14:01:10.880260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880310] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=4673293876, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:10.880339] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.880475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.880938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.881162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.881167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.881446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.881453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.881695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.881701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.881941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.881946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.882176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.882182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.882414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.882419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.882634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.882652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.882889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.882895] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.883132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.883137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.883352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.883357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.883573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.883578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.883791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.883819] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.884052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.884058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.884045] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:686) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] Cache replace map node details(ret=0, replace_node_count=0, replace_time=3621, replace_start_pos=817882, replace_num=31457) [2025-04-03 14:01:10.884075] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=10) [2025-04-03 14:01:10.884279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.884284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.884878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.884886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.885875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.886921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887102] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887229] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887943] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.887950] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888651] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.888921] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:10.888942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=0][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:10.888952] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1545-0-0] [lt=9][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4013, ret=-4013) [2025-04-03 14:01:10.888960] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2025-04-03 14:01:10.888968] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:10.888979] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:10.888986] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] failed to read without check sys variable(ret=-4013, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1001, check_sys_variable=false, snapshot_timestamp=-1) [2025-04-03 14:01:10.888998] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2025-04-03 14:01:10.889088] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] load failed(ret=-4013, for_update=false) [2025-04-03 14:01:10.889095] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] core_table load failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:10.889101] WDIAG [SHARE] get_core_schema_version (ob_global_stat_proxy.cpp:390) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] get failed(ret=-4013) [2025-04-03 14:01:10.889118] WDIAG [SHARE.SCHEMA] get_core_version (ob_schema_service_sql_impl.cpp:857) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] get_core_schema_version failed(ret=-4013) [2025-04-03 14:01:10.889125] WDIAG [SHARE.SCHEMA] can_read_schema_version (ob_schema_service_sql_impl.cpp:6676) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] failed to get core schema version(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", schema_status={tenant_id:1001, snapshot_timestamp:-1, readable_schema_version:-1}) [2025-04-03 14:01:10.889134] WDIAG [SHARE.SCHEMA] fetch_schema (ob_schema_cache.cpp:939) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] incremant schema is not readable now, waiting and retry(ret=-4013, retry_times=0, schema_version=1) [2025-04-03 14:01:10.889142] WDIAG [SHARE.SCHEMA] get_schema (ob_multi_version_schema_service.cpp:636) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] fetch schema failed(tenant_id=1001, schema_type=15, schema_id=1001, schema_version=1, ret=-4013) [2025-04-03 14:01:10.889152] WDIAG [SHARE.SCHEMA] get_schema (ob_schema_getter_guard.cpp:4562) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] get schema failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", schema_status={tenant_id:1001, snapshot_timestamp:-1, readable_schema_version:-1}, schema_type=15, schema_id=1001, schema_version=1, specified_version=-1) [2025-04-03 14:01:10.889161] WDIAG [SHARE.SCHEMA] get_sys_variable_schema (ob_schema_getter_guard.cpp:434) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] fail to get sys var schema(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001) [2025-04-03 14:01:10.889167] WDIAG [SQL.SESSION] get_global_sys_variable (ob_basic_session_info.cpp:1003) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] get sys variable schema failed(ret=-4013) [2025-04-03 14:01:10.889175] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:281) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] failed to get global sys variable(ret=-4013, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-04-03 14:01:10.889186] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:74) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] failed to get mem limit(ret=-4013, tenant_id=1001, mem_limit=53687091, config_mem_limit_=16777216) [2025-04-03 14:01:10.889193] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:185) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=7] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=2523, mem_used=10398720) [2025-04-03 14:01:10.903799] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.903996] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=176][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.904076] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=59][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.904128] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:10.938493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.938608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=113] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.939884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.939906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.939966] WDIAG [LIB] ~Guard (ob_malloc_time_monitor.cpp:34) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] [MALLOC_TIME_MONITOR](cost_time=447662, time_guard=time dist: ALLOC_CHUNK_END=2, WASH_KVCACHE_END=30504, ALLOC_CHUNK_END=44, WASH_KVCACHE_END=48041, ALLOC_CHUNK_END=25, WASH_KVCACHE_END=35588, ALLOC_CHUNK_END=25, WASH_KVCACHE_END=8569, ALLOC_CHUNK_END=21, WASH_KVCACHE_END=10154, ALLOC_CHUNK_END=26, WASH_KVCACHE_END=13004, ALLOC_CHUNK_END=24, WASH_KVCACHE_END=17370, ALLOC_CHUNK_END=18195, WASH_KVCACHE_END=7868, attr=tenant_id=1004, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, size=16256) [2025-04-03 14:01:10.940008] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4013] allocate memory fail(attr=tenant_id=1004, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:10.940018] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:10.940072] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=447791) [2025-04-03 14:01:10.940095] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=447802) [2025-04-03 14:01:10.940104] WDIAG [STORAGE.TRANS] post_cluster_heartbeat_rpc_ (ob_tenant_weak_read_service.cpp:911) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] post cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_service_master="172.20.0.3:2882", tenant_id_=1004, req={req_server:"172.20.0.5:2882", version:{val:1743659606093488000, v:0}, valid_part_count:0, total_part_count:4, generate_timestamp:1743660070492280}, cluster_service_tablet_id={id:226}) [2025-04-03 14:01:10.940190] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=83][errcode=-4013] tenant weak read service cluster heartbeat RPC fail(ret=-4013, rcode={code:-4013, msg:"post cluster heartbeat rpc failed, tenant_id=1004", warnings:[]}, tenant_id_=1004, dst="172.20.0.5:2882", cluster_service_tablet_id={id:226}) [2025-04-03 14:01:10.940241] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4013] post cluster heartbeat rpc fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1004, local_server_version={val:1743659606093488000, v:0}, valid_part_count=0, total_part_count=4, generate_timestamp=1743660070492280) [2025-04-03 14:01:10.940270] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] tenant weak read service do cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1004, last_post_cluster_heartbeat_tstamp_=1743660070338520, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="172.20.0.3:2882") [2025-04-03 14:01:10.940283] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4389] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1004, do_cluster_heartbeat' cost too much time, used=448005, time_dist: get_server_version=2, post_cluster_heartbeat_rpc=447986) [2025-04-03 14:01:10.940997] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=4595516452, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:10.941072] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=60] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=5, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.941481] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=46] get wrs ts(ls_id={id:1001}, delta=4793585773, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:10.941532] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=40] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=5, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.942174] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=51] get wrs ts(ls_id={id:1002}, delta=4805873029, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:10.942228] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=5, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.942701] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:1743655258634689680, v:0}, this={ls_id_:{id:1003}, role_:2, proposal_id_:13, palf_committed_end_lsn_:{lsn:74671177}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1743655258634689680, v:0}}) [2025-04-03 14:01:10.942748] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=38] get wrs ts(ls_id={id:1003}, delta=4591459099, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:10.942763] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=4, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.942876] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1004, thread_timer_task' cost too much time, used=450605, time_dist: do_cluster_heartbeat=448022) [2025-04-03 14:01:10.907223] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=31700][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:10.943121] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=4595518791, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:10.943405] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:191) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=7] release old cannot free more memory [2025-04-03 14:01:10.943150] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.943484] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=287] get wrs ts(ls_id={id:1001}, delta=4793587925, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:10.943504] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.943492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=31899] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.943525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.927053] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=39471][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=61149) [2025-04-03 14:01:10.943576] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16519][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=77680) [2025-04-03 14:01:10.943603] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660070865906], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:10.943631] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660070865906], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:10.943656] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660070865906]}) [2025-04-03 14:01:10.943884] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1002}, delta=4805875181, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:10.943901] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.943902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.943937] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.943981] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1003}, delta=4591460176, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:10.944021] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:10.944670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.944698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.912421] INFO [LIB] operator() (ob_malloc_allocator.cpp:567) [17419][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=32318] [MEMORY] tenant: 500, limit: 9,223,372,036,854,775,807 hold: 8,092,700,672 rpc_hold: 0 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 486,363,136 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= GLIBC hold_bytes= 4,153,344 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= CO_STACK hold_bytes= 81,788,928 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LIBEASY hold_bytes= 3,350,528 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 8,388,608 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 7,733,248 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 7,382,781,952 limit= 9,223,372,036,854,775,807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 118,140,928 limit= 9,223,372,036,854,775,807 [2025-04-03 14:01:10.945133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.945158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.945252] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=35755][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:10.945347] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:10.945373] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] oops, alloc failed, tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=9223372036854775807, tenant_hold=8096894976, tenant_limit=9223372036854775807 [2025-04-03 14:01:10.945393] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:10.945407] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:10.945421] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660070943680], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:10.945457] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660070943680], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:10.945479] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660070943680]}) [2025-04-03 14:01:10.945474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.945531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=55] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.945715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.945729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.945932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.945944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F1-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:10.945940] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:176) [17419][MemDumpTimer][T0][Y0-0000000000000000-0-0] [lt=32658] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 486,363,136 used= 465,756,976 limit= 9,223,372,036,854,775,807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 63 washed_blocks= 428 washed_size= 69,197,824 [MEMORY] hold= 134,238,416 used= 134,217,736 count= 2 avg_used= 67,108,868 block_cnt= 2 chunk_cnt= 2 mod=CACHE_MAP_BKT [MEMORY] hold= 52,517,184 used= 50,996,736 count= 3,317 avg_used= 15,374 block_cnt= 3,317 chunk_cnt= 33 mod=TZInfoArray [MEMORY] hold= 33,574,912 used= 33,554,464 count= 1 avg_used= 33,554,464 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 31,477,760 used= 31,458,304 count= 1 avg_used= 31,458,304 block_cnt= 1 chunk_cnt= 1 mod=ash_list [MEMORY] hold= 19,070,976 used= 19,051,808 count= 1 avg_used= 19,051,808 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 18,894,848 used= 18,874,392 count= 1 avg_used= 18,874,392 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 17,199,104 used= 17,180,640 count= 1 avg_used= 17,180,640 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MB_HANDLE [MEMORY] hold= 12,861,440 used= 12,340,200 count= 157 avg_used= 78,600 block_cnt= 157 chunk_cnt= 27 mod=[T]ObSessionDIB [MEMORY] hold= 12,484,608 used= 12,478,464 count= 6 avg_used= 2,079,744 block_cnt= 6 chunk_cnt= 6 mod=Compressor [MEMORY] hold= 10,776,576 used= 10,735,648 count= 11 avg_used= 975,968 block_cnt= 11 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 5,312,480 used= 5,269,720 count= 15 avg_used= 351,314 block_cnt= 13 chunk_cnt= 10 mod=DeadLock [MEMORY] hold= 5,087,232 used= 5,027,008 count= 6 avg_used= 837,834 block_cnt= 6 chunk_cnt= 4 mod=KvstCachWashStr [MEMORY] hold= 4,747,072 used= 4,726,520 count= 2 avg_used= 2,363,260 block_cnt= 2 chunk_cnt= 2 mod=HashBucTenComMo [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=PxP2PDhMgrKey [MEMORY] hold= 4,739,072 used= 4,718,712 count= 1 avg_used= 4,718,712 block_cnt= 1 chunk_cnt= 1 mod=HashPxBlooFilKe [MEMORY] hold= 4,473,088 used= 4,367,104 count= 348 avg_used= 12,549 block_cnt= 188 chunk_cnt= 21 mod=TenantCtxAlloca [MEMORY] hold= 4,386,768 used= 4,293,088 count= 53 avg_used= 81,001 block_cnt= 29 chunk_cnt= 11 mod=PartitTableTask [MEMORY] hold= 4,307,456 used= 4,278,711 count= 3 avg_used= 1,426,237 block_cnt= 3 chunk_cnt= 3 mod=SqlDtlMgr [MEMORY] hold= 4,232,192 used= 4,203,008 count= 6 avg_used= 700,501 block_cnt= 4 chunk_cnt= 4 mod=BaseLogWriter [MEMORY] hold= 4,214,880 used= 4,194,328 count= 2 avg_used= 2,097,164 block_cnt= 2 chunk_cnt= 2 mod=SerFuncRegHT [MEMORY] hold= 4,046,848 used= 3,969,120 count= 13 avg_used= 305,316 block_cnt= 13 chunk_cnt= 7 mod=DedupQueue [MEMORY] hold= 3,276,704 used= 3,220,064 count= 8 avg_used= 402,508 block_cnt= 8 chunk_cnt= 7 mod=Omt [MEMORY] hold= 3,181,824 used= 3,154,112 count= 19 avg_used= 166,005 block_cnt= 6 chunk_cnt= 5 mod=RpcBuffer [MEMORY] hold= 3,166,208 used= 3,145,744 count= 1 avg_used= 3,145,744 block_cnt= 1 chunk_cnt= 1 mod=CACHE_MAP_LOCK [MEMORY] hold= 2,756,096 used= 2,513,728 count= 1,267 avg_used= 1,984 block_cnt= 425 chunk_cnt= 38 mod=CreateContext [MEMORY] hold= 2,490,368 used= 2,472,736 count= 38 avg_used= 65,072 block_cnt= 38 chunk_cnt= 22 mod=LatchStat [MEMORY] hold= 2,383,872 used= 2,097,152 count= 35 avg_used= 59,918 block_cnt= 35 chunk_cnt= 19 mod=[T]char [MEMORY] hold= 2,379,776 used= 2,359,608 count= 1 avg_used= 2,359,608 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdUnitMa [MEMORY] hold= 2,167,552 used= 2,129,912 count= 7 avg_used= 304,273 block_cnt= 7 chunk_cnt= 5 mod=FixedQueue [MEMORY] hold= 2,113,536 used= 2,097,216 count= 2 avg_used= 1,048,608 block_cnt= 2 chunk_cnt= 2 mod=ConcurHashMap [MEMORY] hold= 2,113,536 used= 2,097,216 count= 2 avg_used= 1,048,608 block_cnt= 2 chunk_cnt= 2 mod=TZInfoMgrMap [MEMORY] hold= 2,015,232 used= 1,963,776 count= 12 avg_used= 163,648 block_cnt= 12 chunk_cnt= 7 mod=Iteratorget_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:11.021059] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021082] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=63][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:11.021096] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:11.021113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021110] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:11.021117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021125] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:11.021170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021174] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=19, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:11.021190] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=41] will sleep(sleep_us=19000, remain_us=4752696, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4283, timeout_timestamp=1743660075773884) [2025-04-03 14:01:11.021226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021778] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021852] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.021954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022064] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022301] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1F5-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.022595] INFO [STORAGE] runTimerTask (ob_ddl_heart_beat_task.cpp:49) [17414][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0] send to rs all task status succeed [2025-04-03 14:01:11.034341] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=9604][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.034591] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17808][T1_RFLWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.034612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17808][T1_RFLWorker][T1][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.034619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17808][T1_RFLWorker][T1][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.034692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17808][T1_RFLWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.034703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17808][T1_RFLWorker][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.040310] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=35][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:19, local_retry_times:19, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:11.040391] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=57][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:11.040406] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:11.040415] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:11.040499] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.040550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.040576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.040672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.040696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.040860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.040883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.040981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041001] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041112] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041667] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.041928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042027] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042064] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042285] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042788] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=51] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.042928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.043284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.043309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.043429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.043446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.043640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.043655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.043781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.043797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044001] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044693] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044708] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.044998] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045173] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045557] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.045803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046351] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.046968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.047104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.047119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.047268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.047282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.047613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.047639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.047901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.047920] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.048140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.048156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.048275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.048290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.048651] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.048669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.048829] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.048844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049004] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=48] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.049982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050126] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050527] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.050557] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:11.050657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050861] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.050877] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051007] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051027] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051174] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051177] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:47) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] refresh tenant config success(tenant_id=1003, *this={last_refresh_ts:1743660071050149, can_tenant_elr:true}) [2025-04-03 14:01:11.051186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051258] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051753] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660071051333]}) [2025-04-03 14:01:11.051787] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:171) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", gts_tenant_info={v:1}) [2025-04-03 14:01:11.051801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051822] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.051998] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.052018] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=42] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052097] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:1388, tid:17422}]) [2025-04-03 14:01:11.052189] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.052213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052224] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.052245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052379] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.052393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052450] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.052485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052494] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.052513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052618] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.052638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052646] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052677] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.052701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052793] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.052820] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052827] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052893] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.052907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052893] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.052913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052922] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.052990] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.053010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053124] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.053149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053230] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=10223) [2025-04-03 14:01:11.053253] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=10244) [2025-04-03 14:01:11.053260] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.053275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053405] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.053419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053449] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4595628824, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:11.053486] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.053491] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.053517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053555] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.053572] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.053596] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.053704] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660071051213]}) [2025-04-03 14:01:11.053720] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:11.053755] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660071051803]}) [2025-04-03 14:01:11.053840] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1001}, delta=4793698044, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:11.053873] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.053959] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1002}, delta=4805985300, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:11.053978] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.054080] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1003}, delta=4591570295, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:11.054122] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.054422] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4389] destruct(*this=time guard '[WRS] [TENANT_WEAK_READ_SERVICE] tenant_id=1004, thread_timer_task' cost too much time, used=11437, time_dist: do_cluster_heartbeat=10294) [2025-04-03 14:01:11.056642] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:11.056685] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28300 [2025-04-03 14:01:11.056756] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28300 [2025-04-03 14:01:11.056781] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:11.058113] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.058157] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660071053787]}) [2025-04-03 14:01:11.059286] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:11.059316] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=28][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:11.060136] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:11.060149] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:11.060477] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=30][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.060514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.060526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.060536] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:11.060544] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:11.060553] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=8][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:11.060672] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:11.060683] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=11][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:11.060680] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.060690] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=6][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.4:2882") [2025-04-03 14:01:11.060704] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.3:2882") [2025-04-03 14:01:11.060704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.060709] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=5][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.5:2882") [2025-04-03 14:01:11.060713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.060721] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=5][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.060742] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.060749] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=6] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:11.060766] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:11.060781] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=5] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071060780, replica_locations:[]}) [2025-04-03 14:01:11.060816] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=23][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.060825] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=8][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071051731}) [2025-04-03 14:01:11.060835] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=8][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.060840] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA670-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.060913] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.060934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.060941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061074] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.061088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061227] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.061242] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061360] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.061379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061387] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061518] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660071059927]}) [2025-04-03 14:01:11.061557] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.061578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061586] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061792] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.061820] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.061828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062005] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.062028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062224] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.062250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062259] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062425] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.062441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062608] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.062630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062734] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.062754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062945] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.062960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.062968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063128] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.063144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063248] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.063268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063364] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.063378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063403] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063506] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:11.063520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.063631] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4805390304, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:11.063652] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.067874] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=69][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.074102] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.074128] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.074138] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.075654] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.077856] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.077880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.077900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078007] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078021] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078124] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078213] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078308] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078394] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078481] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078607] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078674] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078820] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078914] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.078928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.078998] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.079017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079124] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.079138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079266] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.079281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079353] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.079384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079467] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:11.079480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.079574] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4673493259, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:11.079600] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.084200] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=9) [2025-04-03 14:01:11.095422] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [18023][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=3061] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b731e9e82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-04-03 14:01:11.098858] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=2652] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:11.103989] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.104013] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.104024] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.104047] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.114274] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.126369] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.126430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=57] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.126445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.126459] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=13][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.126659] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.126695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.126715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.126802] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.126856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=51] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.126867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.126952] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.126989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127101] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.127141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127236] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.127273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127366] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.127399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127409] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127489] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.127517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127619] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.127647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127758] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.127789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127800] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127886] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.127916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.127926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128067] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.128104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128184] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.128204] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128307] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.128328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128339] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128424] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.128461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128569] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.128600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128697] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.128730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128919] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.128948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.128968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.129103] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.129130] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.129142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.129294] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.129329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.129340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.129453] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.129488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.129500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.129560] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.129592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.129603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.133629] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=38][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:11.133668] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=38][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660069918552], now0=[mts=1743660069918552], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:11.133701] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=29][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719464}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660069917558, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:11.133777] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=69][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:11.133835] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=56][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719464}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660069917558, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:11.133905] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=66] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f7dce008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660069865723, tx_desc={this:0x2b7399219100, tx_id:{txid:47719464}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660069917558, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660054039470, use_das=false, nested_level=0, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}, plan=0x2b73976ba050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:11.134024] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=116][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:11.134065] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=37][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:11.134079] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:11.134092] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:11.134114] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=19][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:11.134137] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=13, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:11.134159] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=17] will sleep(sleep_us=13000, remain_us=12905302, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=13, v.err_=-4283, timeout_timestamp=1743660084039460) [2025-04-03 14:01:11.143753] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1}, delta=4595719794, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:11.143831] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=48] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.143956] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=38] get wrs ts(ls_id={id:1001}, delta=4793787851, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:11.143989] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.144087] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1002}, delta=4806075107, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:11.144116] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.144425] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=46] get wrs ts(ls_id={id:1003}, delta=4591660102, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:11.144468] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.147278] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:13, local_retry_times:13, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:11.147348] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=44][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:11.147365] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:11.147399] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=32][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:11.147439] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=24][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:11.147527] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [18108][T1002_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=1419] BlockGCTimerTask success(ret=0, cost_time_us=35, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.20.0.5:2882", log_dir:"/data/oceanbase_data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):1536, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):1536, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):315, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-04-03 14:01:11.147743] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.147772] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.147798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.147876] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=76][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.148085] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.148113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.148126] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.148285] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.148310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.148321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.148487] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.148527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.148540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.148698] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67292 | score=3.557441 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3976 | score=0.711488 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2529 | get_cnt= 9315 | score=2588.518738 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646304 | score=86585.883631 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1910 | score=677.417329 | [2025-04-03 14:01:11.148732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.148745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.153728] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.153969] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.154012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.154028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.154248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.154275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.154479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.154516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.154920] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.154948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.155269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.155294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.155487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.155513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.155831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.155856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.155971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.155994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.156892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.157183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.157214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.157723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.157748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.158705] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.158735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.158934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.158954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.159306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.159329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.159475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.159495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.159674] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.159702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.159877] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.159916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.160408] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.160426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.160920] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=4805487642, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:11.160973] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=46] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.161460] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=29] [WRS] update tenant weak read server version(tenant_id=1003, server_version={version:{val:1743659951195473000, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1743660071160456}, version_delta=119965986) [2025-04-03 14:01:11.162082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162254] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.162631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.164683] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660071163992]}) [2025-04-03 14:01:11.165329] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660071164726]}) [2025-04-03 14:01:11.166644] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660071165371]}) [2025-04-03 14:01:11.167954] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.169569] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660071166693]}) [2025-04-03 14:01:11.171553] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.171608] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=52][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.171649] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=38] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:11.171686] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660071169996]}) [2025-04-03 14:01:11.171775] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=18] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:11.172003] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=27] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071172000, replica_locations:[]}) [2025-04-03 14:01:11.172163] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=154][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.172235] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=69][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071164645}) [2025-04-03 14:01:11.172288] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=51][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.172300] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA671-0-0] [lt=11][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.174234] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.174294] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.174312] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.175730] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.177908] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=4673591762, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:11.177939] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.182335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182661] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182881] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.182916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183022] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183072] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=49] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183360] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.183379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183400] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=35737) [2025-04-03 14:01:11.183417] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=35752) [2025-04-03 14:01:11.183433] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:11.183583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.183981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184004] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184083] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F6-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.184405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184474] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.184497] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=55647) [2025-04-03 14:01:11.184511] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=13][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=55659) [2025-04-03 14:01:11.184590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184962] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.184981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185341] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=40] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.185975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.186124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.186160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.186307] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.186326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.186657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.186683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.187467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.187491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.187672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.187694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.187769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.187786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.187945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.187965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.188549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.188696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.188856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.189356] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.189570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.189937] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.189980] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=40][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.189996] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:11.190052] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=25] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:11.190099] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:582) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=11] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-04-03 14:01:11.190126] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:221) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D7-0-0] [lt=22] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=308) [2025-04-03 14:01:11.204131] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.204175] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=37] re_register_parent reason(palf_id=1002, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660067195097, last_first_register_time_us=1743648247019455, register_time_us=1743660067195097, curr_time_us=1743660071204173) [2025-04-03 14:01:11.207052] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="172.20.0.5:2882") [2025-04-03 14:01:11.207093] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.207119] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.207137] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.214371] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.220819] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.220844] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=24][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.220854] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:11.220978] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:11.221192] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=17] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071221190, replica_locations:[]}) [2025-04-03 14:01:11.221228] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=33][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.221250] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=21][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071218830}) [2025-04-03 14:01:11.221262] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=11][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.221271] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA672-0-0] [lt=8][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.221866] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 577.00; ) [2025-04-03 14:01:11.221911] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:11.222016] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:11.234246] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE371-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.243305] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.243522] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.243690] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.243889] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.244004] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.244148] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.244168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.244280] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.244475] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.244638] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.244742] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.244986] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.245123] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.245218] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.245325] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.245427] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.245628] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.245854] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.246012] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.246145] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4595821564, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:11.246173] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.246245] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1001}, delta=4793890699, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:11.246264] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.246479] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1002}, delta=4806177955, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:11.246512] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.246614] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1003}, delta=4591762950, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:11.246634] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.253832] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=53][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.268113] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.269512] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.269550] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1555) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=26] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=25165824, ctx.total_memstore_used_=58720256, ctx.total_memstore_hold_=58720256, memstore_freeze_trigger_limit_=43837840, tenant_id=1001) [2025-04-03 14:01:11.269759] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.269790] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.274292] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.274318] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.274327] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.275606] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA673-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.275634] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA673-0-0] [lt=26][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.275665] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA673-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071275663, replica_locations:[]}) [2025-04-03 14:01:11.275686] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA673-0-0] [lt=19][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.275695] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA673-0-0] [lt=9][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071273649}) [2025-04-03 14:01:11.275704] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA673-0-0] [lt=8][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.275709] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA673-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.275839] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=54][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.277964] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.284298] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.286207] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.304201] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.304225] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.304234] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.304242] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.314460] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.338118] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA674-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.338162] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA674-0-0] [lt=41][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.338199] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA674-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071338197, replica_locations:[]}) [2025-04-03 14:01:11.338277] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA674-0-0] [lt=74][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.338309] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA674-0-0] [lt=30][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071334361}) [2025-04-03 14:01:11.338327] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA674-0-0] [lt=17][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.338336] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA674-0-0] [lt=9][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.349162] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [18240][T1002_TabletShe][T1002][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.353151] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.353301] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [18779][T1_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=2149] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.353892] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.368102] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.374401] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.374427] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.374437] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.375919] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.380375] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA675-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.380444] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA675-0-0] [lt=67][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.380462] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA675-0-0] [lt=16] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:11.380534] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA675-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071380531, replica_locations:[]}) [2025-04-03 14:01:11.380594] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA675-0-0] [lt=55][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.380614] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA675-0-0] [lt=18][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071377836}) [2025-04-03 14:01:11.380634] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA675-0-0] [lt=18][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.380650] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA675-0-0] [lt=15][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.381112] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660071380313]}) [2025-04-03 14:01:11.396113] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.400282] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.402773] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:158) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=0] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1743616801173178850, v:0}, checkpoint_scn_in_ls_meta={val:1743616801173178850, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-04-03 14:01:11.402851] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:13286744064}, ls->get_ls_id()={id:1}) [2025-04-03 14:01:11.404255] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.404280] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.404289] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.404311] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.414232] ERROR issue_dba_error (ob_log.cpp:1875) [18051][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4013, file="ob_multi_version_garbage_collector.cpp", line_no=222, info="repeat refresh failed too much time") [2025-04-03 14:01:11.414260] EDIAG [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:222) [18051][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] repeat refresh failed too much time(ret=-4013, this={this:0x2b731f5c71a0, last_study_timestamp:1743655150668510, last_refresh_timestamp:1743655210680275, last_reclaim_timestamp:1743660011387368, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:true, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1743655084793942101, v:0}, is_inited:true}, current_timestamp=1743660071405397) BACKTRACE:0x125fd785 0x50b6335 0x51fab28 0x51fa63b 0x51fa57e 0x51fa3a6 0xef83d9e 0xef927e9 0x5209517 0x5206c75 0x520639e 0x7662b60 0x128d17e1 0x128cdbbf 0x2b72da6c4ea5 0x2b72da9d7b0d [2025-04-03 14:01:11.414705] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.432613] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.432739] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56224 [2025-04-03 14:01:11.440387] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:11.443684] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.444215] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.444546] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.444987] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.445561] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.445880] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.446156] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.446337] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.446398] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE79-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.446500] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.446761] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.447010] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.447376] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.447549] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.447760] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.447969] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.448392] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.448452] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE79-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.448907] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.449639] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.449751] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.450574] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1002}, delta=4806381304, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:11.450904] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1003}, delta=4591967368, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:11.453977] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.463525] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.468200] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.474468] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.474494] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.474502] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.476182] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.479148] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.484099] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660071483463]}) [2025-04-03 14:01:11.485412] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18504][T1004_TenantInf][T1003][YB42AC140005-000631D6C04D1187-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.488271] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA676-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.488299] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA676-0-0] [lt=26][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.488328] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA676-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071488326, replica_locations:[]}) [2025-04-03 14:01:11.488361] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA676-0-0] [lt=30][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.488375] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA676-0-0] [lt=13][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071482759}) [2025-04-03 14:01:11.488388] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA676-0-0] [lt=12][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.488404] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA676-0-0] [lt=15][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.492384] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.501331] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=93] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:11.504345] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.505141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.505383] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.505420] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.506970] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.507003] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.514854] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.515272] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1][YB42AC140005-000631D6BBFCD1EA-0-0] [lt=18][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.516649] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17414][ServerGTimer][T1][YB42AC140005-000631D6BBFCD1EA-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.532086] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.543740] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4596119984, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:11.543791] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=43] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.544192] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=63] get wrs ts(ls_id={id:1001}, delta=4794188041, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:11.544681] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1002}, delta=4806476379, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:11.549597] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA677-0-0] [lt=8][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.551228] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA677-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.551280] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA677-0-0] [lt=50][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.551346] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA677-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071551344, replica_locations:[]}) [2025-04-03 14:01:11.551427] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA677-0-0] [lt=76][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.551465] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA677-0-0] [lt=37][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071548514}) [2025-04-03 14:01:11.551486] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA677-0-0] [lt=20][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.551499] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA677-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.554050] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.558589] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56234 [2025-04-03 14:01:11.561235] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.561295] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=51] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.568251] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.574554] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.574609] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.574634] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.576275] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.578206] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4673992112, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:11.578272] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=59] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.587655] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17690][T1_Occam][T1][YB42AC140005-000631D6BCCCDA20-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.588581] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17690][T1_Occam][T1][YB42AC140005-000631D6BCCCDA20-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.589392] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA678-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.589451] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA678-0-0] [lt=57][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.589497] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA678-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071589494, replica_locations:[]}) [2025-04-03 14:01:11.589556] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA678-0-0] [lt=55][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.589576] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA678-0-0] [lt=19][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071586803}) [2025-04-03 14:01:11.589598] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA678-0-0] [lt=21][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.589610] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA678-0-0] [lt=11][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.589653] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.590645] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660071589991]}) [2025-04-03 14:01:11.604260] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.604553] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.604606] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.604625] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.604642] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.608111] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18235][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.614943] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.625859] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17558][RpcIO][T0][Y0-0000000000000000-0-0] [lt=2] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.630188] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.636497] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=60] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.644057] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.644408] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.644645] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.644959] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.645207] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.645445] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.645705] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:11.648408] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1003}, delta=4592164411, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:11.654118] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.668318] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.674713] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.674797] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.674844] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.676371] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=67][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.679852] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4674093970, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:11.691437] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660071691057]}) [2025-04-03 14:01:11.691636] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660071691477]}) [2025-04-03 14:01:11.691796] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660071691646]}) [2025-04-03 14:01:11.692415] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660071691805]}) [2025-04-03 14:01:11.693358] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660071692626]}) [2025-04-03 14:01:11.694957] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA679-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.695005] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA679-0-0] [lt=46][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.695061] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA679-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071695059, replica_locations:[]}) [2025-04-03 14:01:11.695118] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA679-0-0] [lt=52][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.695144] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA679-0-0] [lt=24][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071691419}) [2025-04-03 14:01:11.695165] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA679-0-0] [lt=19][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.695178] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA679-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.704625] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.704678] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.704708] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.704728] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.704921] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=31] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:11.706210] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.714978] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.715483] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=26] ====== check clog disk timer task ====== [2025-04-03 14:01:11.715508] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=17] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.724305] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67A-0-0] [lt=12][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.725988] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67A-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.726052] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67A-0-0] [lt=60][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.726118] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67A-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071726116, replica_locations:[]}) [2025-04-03 14:01:11.726176] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67A-0-0] [lt=53][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.726205] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67A-0-0] [lt=27][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071723317}) [2025-04-03 14:01:11.726230] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67A-0-0] [lt=24][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.726242] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67A-0-0] [lt=11][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.731114] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.732978] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.746788] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=39] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.748746] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.751373] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.754185] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.754307] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=17][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:31904, tid:17513}]) [2025-04-03 14:01:11.761092] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4806087618, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:11.761117] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.768377] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.775569] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.775628] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.775645] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.776285] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.776313] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.776401] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.777127] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.777157] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.778098] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.778127] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.778352] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4674191952, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:11.778374] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.778590] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.778616] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.779223] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.779262] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.779326] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=14725] PNIO [ratelimit] time: 1743660071779325, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007637, add_bytes: 0 [2025-04-03 14:01:11.779794] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.779857] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=61][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.780598] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.780623] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.781499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.781532] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.781543] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.782244] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.782268] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.782711] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.782733] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.783213] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.783241] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.783922] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.783946] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.784359] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.784393] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.784532] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.784551] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.784903] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.784930] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.785350] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:11.785387] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:11.785430] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.785461] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=10485) [2025-04-03 14:01:11.785478] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=10499) [2025-04-03 14:01:11.785494] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:4768903583}, prev_lsn={lsn:4768903461}, fetch_start_log_id=30980588, last_slide_log_id=30980587, fetch_log_size=41943040, accepted_mode_pid=450, last_fetch_req_time=1743660071785492, last_fetch_end_lsn={lsn:4810846623}, last_fetch_max_log_id=30982635, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:30980587, max_lsn:4768903583, max_scn:{val:1743655263687025503, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:4768903583}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:4726983076}}, last_submit_lsn:{lsn:4768903461}, last_submit_end_lsn:{lsn:4768903583}, last_submit_log_id:30980587, last_submit_log_pid:450, max_flushed_lsn:{lsn:4768903461}, max_flushed_end_lsn:{lsn:4768903583}, max_flushed_log_pid:450, committed_end_lsn:{lsn:4768903583}, last_slide_log_id:30980587, last_slide_scn:{val:1743655263687025503, v:0}, last_slide_lsn:{lsn:4768903461}, last_slide_end_lsn:{lsn:4768903583}, last_slide_log_pid:450, last_slide_log_accum_checksum:1659985352, last_fetch_end_lsn:{lsn:4810846623}, last_fetch_max_log_id:30982635, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660071785492, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b73485f0530}) [2025-04-03 14:01:11.785627] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=121][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.785661] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.785685] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.786233] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.786283] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=42] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56262 [2025-04-03 14:01:11.786366] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56262 [2025-04-03 14:01:11.786395] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:11.794278] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660071793653]}) [2025-04-03 14:01:11.794716] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660071794318]}) [2025-04-03 14:01:11.794838] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.795423] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660071794747]}) [2025-04-03 14:01:11.795758] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.795778] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.795785] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=6] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:11.795799] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:11.795825] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071795823, replica_locations:[]}) [2025-04-03 14:01:11.795849] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=22][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.795858] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=8][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071794254}) [2025-04-03 14:01:11.795866] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=7][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.795871] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67B-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.796213] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660071795444]}) [2025-04-03 14:01:11.797715] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660071796622]}) [2025-04-03 14:01:11.804715] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47] [PALF_DUMP](palf_id=1, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:29484365, max_lsn:4316157304, max_scn:{val:1743655475525511037, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:4316157304}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:4274214264}}, last_submit_lsn:{lsn:4316157182}, last_submit_end_lsn:{lsn:4316157304}, last_submit_log_id:29484365, last_submit_log_pid:624, max_flushed_lsn:{lsn:4316157182}, max_flushed_end_lsn:{lsn:4316157304}, max_flushed_log_pid:624, committed_end_lsn:{lsn:4316157182}, last_slide_log_id:29484364, last_slide_scn:{val:1743655475423544390, v:0}, last_slide_lsn:{lsn:4316157060}, last_slide_end_lsn:{lsn:4316157182}, last_slide_log_pid:624, last_slide_log_accum_checksum:1508656143, last_fetch_end_lsn:{lsn:4358100222}, last_fetch_max_log_id:29486412, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660071505369, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b736e5ba530}, [StateMgr]={this:0x2b736e5bea30, self:"172.20.0.5:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:627}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:29484365, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2b736e5c52f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246085147, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:624, prev:{version:1, config:{config_version:{proposal_id:624, config_seq:414}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246085147, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:624, config_seq:415}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246085147, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:624, prev_lsn:{lsn:4305589396}, prev_mode_pid:624}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:624, config_seq:415}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660069097322, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660069097322, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b736e5bbc70}, [ModeMgr]={palf_id:1, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:624, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:624, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:624, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:4227600384}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:627}, log_config_meta:{version:3, proposal_id:624, prev:{version:1, config:{config_version:{proposal_id:624, config_seq:414}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246085147, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:624, config_seq:415}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246085147, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:624, prev_lsn:{lsn:4305589396}, prev_mode_pid:624}, log_snapshot_meta:{version:1, base_lsn:{lsn:4227600384}, prev_log_info:{log_id:28971429, lsn:{lsn:4227596178}, scn:{val:1743602955857709777, v:0}, log_proposal_id:624, accum_checksum:1785596109}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:624, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:28672}, readable_log_tail:{lsn:28672}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:4361809920}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1/meta", dir_fd:116, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67076096, block_header_serialize_buf_:0x2b736e5c0bb8, flashback_version:0}, log_storage:{log_tail:{lsn:4316157304}, readable_log_tail:{lsn:4316157304}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:4294705152}, min_scn:{val:1743640537306688823, v:0}, curr_block_id:64, palf_id:1}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1/log", dir_fd:117, min_block_id:63, max_block_id:65, curr_writable_block_id:64}, logical_block_size_:67104768, curr_block_writable_size_:45652616, block_header_serialize_buf_:0x2b736e5c2838, flashback_version:0}, palf_epoch:5, last_purge_throttling_ts:1743648282114161, this:0x2b736e5bf170}, [Reconfirm]={palf_id:1, self:"172.20.0.5:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b736e5bebb0}) [2025-04-03 14:01:11.805230] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=489][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:11.805258] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=26] [PALF_DUMP](palf_id=1, self="172.20.0.5:2882", ack_info_list=[]) [2025-04-03 14:01:11.805286] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:11.805310] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=2][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.805336] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.805354] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.805377] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.815103] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.834564] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56264 [2025-04-03 14:01:11.838279] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.849026] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1003}, delta=4592365164, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:11.850849] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:11.850990] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.852087] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=30] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.854247] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.854566] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4498, tid:18702}]) [2025-04-03 14:01:11.858588] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.860478] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.861291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.861329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.861558] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.861584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.861711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.861733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862917] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.862936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.863164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.863186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.863411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.864070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.864352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.864478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.864633] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.864659] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.864673] WDIAG [STORAGE.TRANS] post_cluster_heartbeat_rpc_ (ob_tenant_weak_read_service.cpp:911) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] post cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_service_master="172.20.0.3:2882", tenant_id_=1003, req={req_server:"172.20.0.5:2882", version:{val:1743659951195473000, v:0}, valid_part_count:0, total_part_count:1, generate_timestamp:1743660071861139}, cluster_service_tablet_id={id:226}) [2025-04-03 14:01:11.864734] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=58][errcode=-4013] tenant weak read service cluster heartbeat RPC fail(ret=-4013, rcode={code:-4013, msg:"post cluster heartbeat rpc failed, tenant_id=1003", warnings:[]}, tenant_id_=1003, dst="172.20.0.5:2882", cluster_service_tablet_id={id:226}) [2025-04-03 14:01:11.864778] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=41][errcode=-4013] post cluster heartbeat rpc fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1003, local_server_version={val:1743659951195473000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1743660071861139) [2025-04-03 14:01:11.864802] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23][errcode=-4013] tenant weak read service do cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1003, last_post_cluster_heartbeat_tstamp_=1743660071664334, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="172.20.0.3:2882") [2025-04-03 14:01:11.864805] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:11.865329] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=53] get wrs ts(ls_id={id:1}, delta=4806191170, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:11.865373] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=36] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.868472] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.872016] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=38][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.873053] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=76][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.875082] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.875123] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.875142] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.876470] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.878530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.878578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.878765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.878802] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.878959] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.878992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879303] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.879982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.880009] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.880158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.880187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.880413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.880613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.881354] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.881398] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.881420] WDIAG [STORAGE.TRANS] post_cluster_heartbeat_rpc_ (ob_tenant_weak_read_service.cpp:911) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] post cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_service_master="172.20.0.3:2882", tenant_id_=1001, req={req_server:"172.20.0.5:2882", version:{val:1743659476426054000, v:0}, valid_part_count:0, total_part_count:1, generate_timestamp:1743660071878416}, cluster_service_tablet_id={id:226}) [2025-04-03 14:01:11.881480] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=57][errcode=-4013] tenant weak read service cluster heartbeat RPC fail(ret=-4013, rcode={code:-4013, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="172.20.0.5:2882", cluster_service_tablet_id={id:226}) [2025-04-03 14:01:11.881512] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4013] post cluster heartbeat rpc fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, local_server_version={val:1743659476426054000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1743660071878416) [2025-04-03 14:01:11.881555] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=-4013] tenant weak read service do cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1743660071679786, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="172.20.0.3:2882") [2025-04-03 14:01:11.881594] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-04-03 14:01:11.884703] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=5) [2025-04-03 14:01:11.885572] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.888637] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.899895] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.900880] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.900917] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.900932] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660071899613], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.900962] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660071899613], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.901007] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660071899613]}) [2025-04-03 14:01:11.902457] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=55][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.902484] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.902497] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660071901073], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.902521] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660071901073], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.902558] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660071901073]}) [2025-04-03 14:01:11.902653] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=5][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.902678] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=24][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.902692] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=10][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:11.902714] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=21][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.4:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:11.903073] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.903361] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=83] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:11.903651] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.903678] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.903690] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660071902584], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.903726] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660071902584], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.903760] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660071902584]}) [2025-04-03 14:01:11.903971] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.903998] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.904010] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=9][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:11.904052] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=40][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.3:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:11.904794] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.904846] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.904863] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.904905] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.905080] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.905105] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.905117] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660071903782], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.905137] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660071903782], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.905159] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660071903782]}) [2025-04-03 14:01:11.905622] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.905653] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=30][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.905666] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=10][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:11.905686] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=19][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.5:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:11.906127] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.906165] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=36][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.906178] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:11.906210] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=20] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:11.906239] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=16] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071906237, replica_locations:[]}) [2025-04-03 14:01:11.906291] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=48][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.906334] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=41][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071900985}) [2025-04-03 14:01:11.906357] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=22][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.906368] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67C-0-0] [lt=10][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.906866] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.906883] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.906892] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660071905499], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.906910] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660071905499], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:11.906926] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660071905499]}) [2025-04-03 14:01:11.914910] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.915261] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:11.916574] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.927210] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.927801] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.938713] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3AE5-0-0] [lt=947][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.941690] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.943933] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4596519928, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:11.943993] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=38] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.944270] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=77] get wrs ts(ls_id={id:1001}, delta=4794587985, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:11.944299] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.944360] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.944476] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1002}, delta=4806875241, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:11.944501] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.944693] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=34] get wrs ts(ls_id={id:1003}, delta=4592461315, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:11.944733] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.946856] WARN alloc_chunk (achunk_mgr.cpp:326) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] No memory or reach tenant memory limit(msg="oops, over total memory limit", hold=10735362048, limit=10737418240) [2025-04-03 14:01:11.947932] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.947985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=48] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.948004] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.948462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.948483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.949020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.949065] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=42] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.949384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.949400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.949915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.949953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.950597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.950625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.951328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.951352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.951942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.951967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.952492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.952512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.953201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.953228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.953797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.953843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=44] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.954205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.954224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.954302] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=21] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660067769299, last_first_register_time_us=1743648246543257, register_time_us=1743660067769299, curr_time_us=1743660071954298) [2025-04-03 14:01:11.954895] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=38][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.954924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.954954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.955400] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.955414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.955447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.955893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.955912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.956294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.956318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.956578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.956602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.956691] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.957563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.957585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.957951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.957966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.958220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.958233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.958857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.958874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.959178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.959190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.959677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.959689] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.959887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.959901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.959898] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.959932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.959940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960066] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960189] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.960928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.961058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.961078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.961171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.961181] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.961286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.961296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.961299] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1}, delta=4806287069, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:11.961328] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:11.961619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.961651] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962126] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.962957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.963125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.963147] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.963206] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:11.963274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.963288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.963509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.963532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.963565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.963575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.964058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.964073] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.964250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.964265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.964722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.964740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.964917] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.964932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965333] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=5][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.965354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965361] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965829] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.965915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966741] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966789] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.966831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.966832] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=40][errcode=-4013] oops, alloc failed, tenant_id=1003, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=1073741800, tenant_hold=441917440, tenant_limit=1073741824 [2025-04-03 14:01:11.966877] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=36][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_LOG_REGISTER_PARENT_REQ, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.966900] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=21][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:11.966919] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=12567) [2025-04-03 14:01:11.966935] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=15][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=12580) [2025-04-03 14:01:11.966956] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] LogRpc post_request failed(ret=-4013, palf_id_=1, req={child:{server:"172.20.0.5:2882", region:"default_region", register_time_us:1743660071954353, keepalive_ts:-1}, is_to_leader:true}, server="172.20.0.3:2882") [2025-04-03 14:01:11.966992] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=34] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.20.0.5:2882") [2025-04-03 14:01:11.967016] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.967085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967321] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.967439] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.967842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968363] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:11.968519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968715] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1}]) [2025-04-03 14:01:11.968743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.968743] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.969065] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969534] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.969655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.970082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.970090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.970388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.970397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.970743] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.970754] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=10][errcode=-4013] oops, alloc failed, tenant_id=1001, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=397373440, tenant_limit=536870912 [2025-04-03 14:01:11.970765] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=7][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.970771] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6][errcode=-4013] rpc memory pool alloc memory failed(sz=1330, alloc_sz=16256) [2025-04-03 14:01:11.970777] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1146) [2025-04-03 14:01:11.970783] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=5][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:11.970794] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=5][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=1981834, timeout_timestamp=1743660073943567) [2025-04-03 14:01:11.970830] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=35][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:11.970840] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=7][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:11.970846] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=5][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:11.970853] WDIAG [SQL] open (ob_result_set.cpp:158) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:11.970860] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=5][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:11.970867] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2025-04-03 14:01:11.970878] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=10][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:11.970897] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=16][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:11.970925] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=16][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:11.970930] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=5][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:11.970940] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=9][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:11.971547] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.971559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972727] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.972786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973048] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973317] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.973837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974195] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.974857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975172] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.975208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975210] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.975220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975228] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.975494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975502] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.975971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.976269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.976284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.976334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.976340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.976404] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:11.976413] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=8][errcode=-4013] oops, alloc failed, tenant_id=1001, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=397373440, tenant_limit=536870912 [2025-04-03 14:01:11.976423] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:11.976440] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=16][errcode=-4013] rpc memory pool alloc memory failed(sz=1305, alloc_sz=16256) [2025-04-03 14:01:11.976445] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=5][errcode=-4013] alloc buffer fail(payload_sz=1121) [2025-04-03 14:01:11.976451] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=5][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:11.976462] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=29999823, timeout_timestamp=1743660101971836) [2025-04-03 14:01:11.976470] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=7][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:11.976479] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:11.976485] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:11.976550] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:11.976730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.976738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.976962] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.976969] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.977980] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.978142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.978501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.978642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.978805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.979154] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.979555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.979941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.980488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.980603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.980745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.980908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.981411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.981851] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67D-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.981866] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67D-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:11.981923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.982100] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67D-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660071982099, replica_locations:[]}) [2025-04-03 14:01:11.982126] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67D-0-0] [lt=24][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:11.982136] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67D-0-0] [lt=9][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660071979176}) [2025-04-03 14:01:11.982145] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67D-0-0] [lt=8][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:11.982150] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67D-0-0] [lt=5][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:11.982207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.982533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.983099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.983363] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.983674] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.984853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.985269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.985302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.986357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.986978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.991360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.992371] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:11.993617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.993641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.994408] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.994430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.994776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:11.996986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1546-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.004188] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1962) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=10] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-04-03 14:01:12.004209] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1464) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=18] start lib cache evict(tenant_id=1002, mem_hold=9027584, mem_limit=26843545, cache_obj_num=34, cache_node_num=34) [2025-04-03 14:01:12.004218] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1481) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=7] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=9027584, mem_limit=26843545, cache_obj_num=34, cache_node_num=34) [2025-04-03 14:01:12.004225] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2694) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=6] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:12.004865] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.004917] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.004940] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.004959] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.006767] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2449) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=5] Dumping All Cache Objs(alloc_obj_list.count()=34, alloc_obj_list=[{obj_id:1, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175493}, {obj_id:2, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176543}, {obj_id:10, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175828}, {obj_id:11, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180892}, {obj_id:23, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181961}, {obj_id:43, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187646}, {obj_id:44, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:45, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188001}, {obj_id:46, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185069}, {obj_id:47, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187994}, {obj_id:48, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185011}, {obj_id:49, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191261}, {obj_id:50, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186550}, {obj_id:51, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182609}, {obj_id:52, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209465}, {obj_id:56, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191136}, {obj_id:57, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182721}, {obj_id:58, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183848}, {obj_id:60, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253856}, {obj_id:64, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176578}, {obj_id:65, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173048}, {obj_id:66, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174057}, {obj_id:67, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183126}, {obj_id:68, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178338}, {obj_id:69, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244258}, {obj_id:70, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224434}, {obj_id:71, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220628}, {obj_id:72, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274876}, {obj_id:73, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214173}, {obj_id:74, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212788}, {obj_id:75, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205532}, {obj_id:76, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196291}, {obj_id:77, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190887}, {obj_id:78, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176654}]) [2025-04-03 14:01:12.006861] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2702) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=92] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:12.007960] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.008003] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660072007377]}) [2025-04-03 14:01:12.008874] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660072008023]}) [2025-04-03 14:01:12.009311] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660072008902]}) [2025-04-03 14:01:12.009830] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660072009324]}) [2025-04-03 14:01:12.010528] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660072010130]}) [2025-04-03 14:01:12.012045] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.012075] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=28][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.012117] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=41] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:12.012141] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:12.012171] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072012162, replica_locations:[]}) [2025-04-03 14:01:12.012205] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=31][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.012217] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=11][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072007988}) [2025-04-03 14:01:12.012234] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=16][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.012243] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67E-0-0] [lt=8][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.015339] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.038765] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.044194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=4] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.044229] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.044421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.044448] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.044612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.044633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.044794] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.044834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.045223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.045247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.045430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.045451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.045641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.045671] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.045846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.045864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.046265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.046288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.046549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.046570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.046886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.046907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.047340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.047363] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.047591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.047613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.048050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.048073] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.048327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.048350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.048978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.049001] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.049381] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-04-03 14:01:12.049447] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=49] get wrs ts(ls_id={id:1}, delta=4596625043, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:12.049475] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.049761] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=47] get wrs ts(ls_id={id:1001}, delta=4794694170, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:12.049793] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.050290] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=50] get wrs ts(ls_id={id:1002}, delta=4806981426, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:12.050327] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.050760] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=30] get wrs ts(ls_id={id:1003}, delta=4592567436, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:12.050798] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.053783] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:103) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=1] tx gc loop thread is running(MTL_ID()=1004) [2025-04-03 14:01:12.054018] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=50] update calc upper info once(calc_upper_info_={min_start_scn_in_ctx_:{val:1743655471770425554, v:0}, keep_alive_scn_:{val:1743655471770425554, v:0}, update_ts_:1743660072054015}) [2025-04-03 14:01:12.054093] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=58] get_min_unreplayed_log_info(lsn={lsn:4316157182}, scn={val:1743655475423544391, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1743660071776575, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:4316157182}, next_to_submit_scn_:{val:1743655475423544391, v:0}, base_lsn_:{lsn:4227600384}, base_scn_:{val:1743616803524366216, v:0}, iterator_:{iterator_impl:{buf_:0x2b738be8d000, next_round_pread_size:2121728, curr_read_pos:38458, curr_read_buf_start_pos:0, curr_read_buf_end_pos:38458, log_storage_:{IteratorStorage:{start_lsn:{lsn:4316118724}, end_lsn:{lsn:4316157182}, read_buf:{buf_len_:487424, buf_:0x2b738be8d000}, block_size:67104768, log_storage_:0x2b736e5c1bf0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1743655475423544390, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1743655475423544390, v:0}, data_checksum:2923900223, flag:1}}, init_mode_version:0, accumulate_checksum:1508656143, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1743655475423544390, v:0}}}}}) [2025-04-03 14:01:12.054959] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:17918, tid:18619}]) [2025-04-03 14:01:12.056048] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=59][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.056351] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=106] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616943687716218, v:0}, clog_base_lsn:{lsn:4227600384}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743618994719735223, v:0}, all_id_meta:{id_meta:[{limited_id:1743655491736477739, latest_log_ts:{val:1743655475118797847, v:0}}, {limited_id:22000001, latest_log_ts:{val:1743616321186417451, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:624, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:12.056468] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=108] update calc upper info once(calc_upper_info_={min_start_scn_in_ctx_:{val:1743655276442494259, v:0}, keep_alive_scn_:{val:1743655276442494259, v:0}, update_ts_:1743660072056466}) [2025-04-03 14:01:12.058580] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1743616894738777400, v:0}, clog_base_lsn:{lsn:3489447936}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616894738777399, v:0}, all_id_meta:{id_meta:[{limited_id:1740557836670287627, latest_log_ts:{val:1740557820032434136, v:0}}, {limited_id:19000001, latest_log_ts:{val:1740413613052489362, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743590092673837460, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:637, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:12.058671] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=86] update calc upper info once(calc_upper_info_={min_start_scn_in_ctx_:{val:1743655263640007290, v:0}, keep_alive_scn_:{val:1743655263640007290, v:0}, update_ts_:1743660072058670}) [2025-04-03 14:01:12.060458] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=15] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1743616823359480140, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616823359480139, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743590092742141696, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:1, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:12.060535] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=73] update calc upper info once(calc_upper_info_={min_start_scn_in_ctx_:{val:1743655476744925757, v:0}, keep_alive_scn_:{val:1743655476744925757, v:0}, update_ts_:1743660072060534}) [2025-04-03 14:01:12.061558] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.061617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=54] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.061638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.061922] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.061966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=41] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.061980] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062141] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.062168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062300] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.062335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062281] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=13] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1003}, ls_create_status:1, clog_checkpoint_scn:{val:1743616894738777400, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616894738777399, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743614587358473541, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:14, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:5505024, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:12.062349] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062473] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.062507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062702] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.062731] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062905] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.062923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.062935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063146] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.063185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=36] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063352] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.063379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063515] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.063542] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063718] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.063755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063767] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.063963] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.064002] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=36] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064154] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.064180] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064313] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.064347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064477] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.064512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064641] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:12.064683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.064872] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4806391615, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:12.064919] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=40] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.068835] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.075256] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.075430] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.075443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075448] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075662] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.075681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075786] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.075797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075888] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.075898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075968] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.075983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.075988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076071] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.076093] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076354] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.076376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076453] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.076464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076597] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.076608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076614] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076655] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.076724] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.076742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.076757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.077165] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.077179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.077185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.077590] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.077611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.077617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.077879] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.077893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.077899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.078479] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.078497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.078503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.078661] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.078672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.078676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.078746] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.078784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.078805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.078985] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:12.078997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079043] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1002, self="172.20.0.5:2882", dest="172.20.0.4:2882", fetch_log_count=2048, fetch_start_lsn={lsn:3578551959}, prev_lsn={lsn:3578551837}, fetch_start_log_id=29401988, last_slide_log_id=29401987, fetch_log_size=41943040, accepted_mode_pid=171, last_fetch_req_time=1743660072079042, last_fetch_end_lsn={lsn:3620494999}, last_fetch_max_log_id=29404035, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1002, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:29401988, max_lsn:3578552081, max_scn:{val:1743655484775020385, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:3578552081}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:3556552704}}, last_submit_lsn:{lsn:3578551959}, last_submit_end_lsn:{lsn:3578552081}, last_submit_log_id:29401988, last_submit_log_pid:171, max_flushed_lsn:{lsn:3578551959}, max_flushed_end_lsn:{lsn:3578552081}, max_flushed_log_pid:171, committed_end_lsn:{lsn:3578551959}, last_slide_log_id:29401987, last_slide_scn:{val:1743655484668882629, v:0}, last_slide_lsn:{lsn:3578551837}, last_slide_end_lsn:{lsn:3578551959}, last_slide_log_pid:171, last_slide_log_accum_checksum:2564257387, last_fetch_end_lsn:{lsn:3620494999}, last_fetch_max_log_id:29404035, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660072079042, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b73b7fc4530}) [2025-04-03 14:01:12.079102] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.079152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079094] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=46] [PALF_DUMP](palf_id=1002, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1002, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:29401988, max_lsn:3578552081, max_scn:{val:1743655484775020385, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:3578552081}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:3556552704}}, last_submit_lsn:{lsn:3578551959}, last_submit_end_lsn:{lsn:3578552081}, last_submit_log_id:29401988, last_submit_log_pid:171, max_flushed_lsn:{lsn:3578551959}, max_flushed_end_lsn:{lsn:3578552081}, max_flushed_log_pid:171, committed_end_lsn:{lsn:3578551959}, last_slide_log_id:29401987, last_slide_scn:{val:1743655484668882629, v:0}, last_slide_lsn:{lsn:3578551837}, last_slide_end_lsn:{lsn:3578551959}, last_slide_log_pid:171, last_slide_log_accum_checksum:2564257387, last_fetch_end_lsn:{lsn:3620494999}, last_fetch_max_log_id:29404035, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660072079042, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b73b7fc4530}, [StateMgr]={this:0x2b73b7fc8a30, self:"172.20.0.5:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:171}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:29401988, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2b73b7fcf2f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246001988, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:171, prev:{version:1, config:{config_version:{proposal_id:171, config_seq:23}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246001988, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:171, config_seq:24}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246001988, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:171, prev_lsn:{lsn:3570698036}, prev_mode_pid:171}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:171, config_seq:24}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660068395453, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660068395453, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b73b7fc5c70}, [ModeMgr]={palf_id:1002, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:171, mode_version:0, access_mode:1, ref_scn:{val:1737450300818354210, v:0}}, accepted_mode_meta:{version:1, proposal_id:171, mode_version:0, access_mode:1, ref_scn:{val:1737450300818354210, v:0}}, last_submit_mode_meta:{version:1, proposal_id:171, mode_version:0, access_mode:1, ref_scn:{val:1737450300818354210, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:3556552704}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:171}, log_config_meta:{version:3, proposal_id:171, prev:{version:1, config:{config_version:{proposal_id:171, config_seq:23}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246001988, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:171, config_seq:24}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246001988, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:171, prev_lsn:{lsn:3570698036}, prev_mode_pid:171}, log_snapshot_meta:{version:1, base_lsn:{lsn:3556552704}, prev_log_info:{log_id:29221225, lsn:{lsn:3556548516}, scn:{val:1743635710828859763, v:0}, log_proposal_id:171, accum_checksum:3696725977}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:171, mode_version:0, access_mode:1, ref_scn:{val:1737450300818354210, v:0}}}, log_meta_storage:{log_tail:{lsn:24576}, readable_log_tail:{lsn:24576}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:3623657472}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1002/1002/meta", dir_fd:107, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67080192, block_header_serialize_buf_:0x2b73b7fcabb8, flashback_version:0}, log_storage:{log_tail:{lsn:3578552081}, readable_log_tail:{lsn:3578552081}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:3556552704}, min_scn:{val:1743635710828859764, v:0}, curr_block_id:53, palf_id:1002}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1002/1002/log", dir_fd:109, min_block_id:53, max_block_id:54, curr_writable_block_id:53}, logical_block_size_:67104768, curr_block_writable_size_:45105391, block_header_serialize_buf_:0x2b73b7fcc838, flashback_version:0}, palf_epoch:7, last_purge_throttling_ts:1743648256074102, this:0x2b73b7fc9170}, [Reconfirm]={palf_id:1002, self:"172.20.0.5:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b73b7fc8bb0}) [2025-04-03 14:01:12.079266] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=167][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:12.079276] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1002, self="172.20.0.5:2882", ack_info_list=[]) [2025-04-03 14:01:12.079283] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:12.079288] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4308) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=4] current log_task status(palf_id=1002, self="172.20.0.5:2882", log_id=29401988, log_task={header:{begin_lsn:{lsn:3578551959}, end_lsn:{lsn:3578552081}, log_id:29401988, min_scn:{val:1743655484775020385, v:0}, max_scn:{val:1743655484775020385, v:0}, data_len:66, proposal_id:171, prev_lsn:{lsn:3578551837}, prev_proposal_id:171, committed_end_lsn:{lsn:3578551959}, data_checksum:1715074492, accum_checksum:465264222, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1743659140753353, freeze_ts:1743659140753354, submit_ts:1743659140753354, flushed_ts:1743659140759424, gen_to_freeze cost time:1, gen_to_submit cost time:1, submit_to_flush cost time:6070}, ack_info_list=[]) [2025-04-03 14:01:12.079304] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.079314] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.079316] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.079349] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079693] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.079722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079901] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.079936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.079946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080263] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.080292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080500] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.080526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080586] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.080620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080689] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.080711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080727] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080792] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.080832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080895] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.080928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080937] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.080982] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.081004] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081112] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.081129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081187] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.081218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081271] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.081292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081301] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081358] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.081380] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081646] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.081684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081833] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:12.081859] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.081870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.082087] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4674495773, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:12.082122] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.084824] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=4) [2025-04-03 14:01:12.104929] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.104954] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.104967] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.104979] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.111535] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.111586] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660072111095]}) [2025-04-03 14:01:12.111606] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:171) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", gts_tenant_info={v:1}) [2025-04-03 14:01:12.112617] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:12.112646] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=26][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:12.113093] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660072111639]}) [2025-04-03 14:01:12.113741] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:12.113759] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=16][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:12.114517] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:12.114533] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=15][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:12.114544] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=10][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:12.114772] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660072113118]}) [2025-04-03 14:01:12.115054] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:12.115074] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=19][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:12.115084] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=8][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.4:2882") [2025-04-03 14:01:12.115093] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=9][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.3:2882") [2025-04-03 14:01:12.115101] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=8][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.5:2882") [2025-04-03 14:01:12.115123] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.115138] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.115147] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:12.115165] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:12.115185] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072115183, replica_locations:[]}) [2025-04-03 14:01:12.115214] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=25][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.115227] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=11][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072111568}) [2025-04-03 14:01:12.115237] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=9][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.115251] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA67F-0-0] [lt=13][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.115412] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.115957] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660072114791]}) [2025-04-03 14:01:12.115985] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660072114791]}) [2025-04-03 14:01:12.117456] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660072116324]}) [2025-04-03 14:01:12.117482] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660072116324]}) [2025-04-03 14:01:12.144194] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=53] get wrs ts(ls_id={id:1}, delta=4596719916, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:12.144223] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.144341] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=4794787973, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:12.144350] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.144497] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1002}, delta=4807075229, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:12.144505] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.144593] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1003}, delta=4592661301, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:12.144600] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=6] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.150562] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=29] ====== check clog disk timer task ====== [2025-04-03 14:01:12.150613] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=36] get_disk_usage(ret=0, capacity(MB):=9216, used(MB):=563) [2025-04-03 14:01:12.150757] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=20] cannot_recycle_log_size statistics(cannot_recycle_log_size=322182499, threshold=2899102924, need_update_checkpoint_scn=false) [2025-04-03 14:01:12.156097] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.161438] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=36] get wrs ts(ls_id={id:1}, delta=4806487846, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:12.161476] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.163742] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.163757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.163762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.163892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.163902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164120] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164212] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164416] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.164845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.168913] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=41] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660068095071, last_first_register_time_us=1743648298762648, register_time_us=1743660068095071, curr_time_us=1743660072168910) [2025-04-03 14:01:12.169080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.169372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.169540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.169706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.170540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.170976] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.175343] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.175396] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.175445] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.176724] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.178832] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=4674593018, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:12.186422] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.186476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=49] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.187496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.188127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.188424] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.189172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.189418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.189451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.190394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.190552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.191214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.192658] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F7-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.197011] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D8-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.200532] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D8-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.200800] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D8-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.203654] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D8-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.203685] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D8-0-0] [lt=29][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.203740] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:582) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D8-0-0] [lt=0] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-04-03 14:01:12.204956] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.204982] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.204992] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.205002] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.210641] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18111][T1002_LogUpdate][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.215453] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.223563] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA680-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.223622] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA680-0-0] [lt=57][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.223660] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA680-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072223658, replica_locations:[]}) [2025-04-03 14:01:12.223710] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA680-0-0] [lt=46][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.223741] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA680-0-0] [lt=30][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072218704}) [2025-04-03 14:01:12.223759] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA680-0-0] [lt=16][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.223772] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA680-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.223890] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=37] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.229692] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA681-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.229721] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA681-0-0] [lt=28][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.229745] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA681-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072229744, replica_locations:[]}) [2025-04-03 14:01:12.229772] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA681-0-0] [lt=25][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.229785] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA681-0-0] [lt=12][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072227408}) [2025-04-03 14:01:12.229795] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA681-0-0] [lt=9][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.229824] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA681-0-0] [lt=28][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.236623] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56270 [2025-04-03 14:01:12.239691] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE372-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.256174] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.257523] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=11][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:12.257563] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=38][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660075703884, now=[mts=1743660071051212], now0=[mts=1743660071051212], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:12.257600] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=33][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a4335d80, tx_id:{txid:76175008}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660071039346, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:12.257682] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=76][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73961321a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a4335d80}) [2025-04-03 14:01:12.257718] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=34][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a4335d80, tx_id:{txid:76175008}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660071039346, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:12.257767] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=46] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.257896] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=127][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:12.257923] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=24][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:12.257936] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:12.257949] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:12.257961] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:12.257980] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=20, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:12.268945] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.275432] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.275487] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.275508] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.276786] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.278080] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:20, local_retry_times:20, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:12.278177] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=73][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:12.278193] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:12.278202] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:12.278660] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.278688] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.278968] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.280916] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.281499] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA682-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.281540] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA682-0-0] [lt=39][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.281596] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA682-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072281594, replica_locations:[]}) [2025-04-03 14:01:12.281645] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA682-0-0] [lt=46][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.281693] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA682-0-0] [lt=45][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072276828}) [2025-04-03 14:01:12.281712] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA682-0-0] [lt=18][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.281724] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA682-0-0] [lt=11][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.284912] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=15] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.285224] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.285255] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:12.287116] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:12.301934] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=9] ====== check clog disk timer task ====== [2025-04-03 14:01:12.301992] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=46] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.305018] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.305087] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=62][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.305145] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.305163] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.306906] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=65] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:12.315546] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.326825] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660072326193]}) [2025-04-03 14:01:12.326851] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=124, get_gts_with_stc_cnt=8266, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-04-03 14:01:12.327495] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660072326873]}) [2025-04-03 14:01:12.328127] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660072327514]}) [2025-04-03 14:01:12.328720] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660072328154]}) [2025-04-03 14:01:12.329246] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.329287] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=38][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.329308] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=19] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:12.329337] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:12.329376] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=24] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072329373, replica_locations:[]}) [2025-04-03 14:01:12.329452] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=72][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.329479] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=25][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072326795}) [2025-04-03 14:01:12.329520] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=38][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.329533] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA683-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.329602] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660072328993]}) [2025-04-03 14:01:12.333563] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:12.333602] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=31] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56274 [2025-04-03 14:01:12.333711] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56274 [2025-04-03 14:01:12.333735] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:12.344451] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4596920545, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:12.344493] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.344611] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1001}, delta=4794988602, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:12.344639] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.344907] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1002}, delta=4807275858, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:12.344939] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.345228] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1003}, delta=4592861936, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:12.345269] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.356251] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.361600] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4806688297, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:12.361630] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.369087] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.375522] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.375579] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.375599] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.376868] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.378964] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-04-03 14:01:12.380799] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=23][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:12.380861] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=61][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660071147662], now0=[mts=1743660071147662], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:12.380887] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=23][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719465}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660071146660, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:12.380964] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=70][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:12.380998] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=32][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719465}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660071146660, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:12.381062] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=61] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f7dce008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660071095929, tx_desc={this:0x2b7399219100, tx_id:{txid:47719465}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660071146660, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660054039470, use_das=false, nested_level=0, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}, plan=0x2b73976ba050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:12.381150] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=86][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:12.381171] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:12.381181] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=9][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:12.381190] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:12.381198] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=6][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:12.381220] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=14, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:12.381249] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=25] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.395312] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=16][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:14, local_retry_times:14, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:12.395411] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=74][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:12.395436] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=23][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:12.395446] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=9][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:12.395473] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:12.398877] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.398934] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:12.400857] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:12.400876] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56276 [2025-04-03 14:01:12.400910] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.405082] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.405108] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.405122] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.405134] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.415586] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.431890] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660072430114]}) [2025-04-03 14:01:12.432446] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:12.432549] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.4:56278 [2025-04-03 14:01:12.433384] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660072431929]}) [2025-04-03 14:01:12.434009] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660072433420]}) [2025-04-03 14:01:12.434496] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.434517] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.434533] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:12.434547] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=8] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:12.434560] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=5] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072434558, replica_locations:[]}) [2025-04-03 14:01:12.434581] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=20][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.434590] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=8][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072431863}) [2025-04-03 14:01:12.434597] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=7][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.434602] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA684-0-0] [lt=5][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.435515] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660072434064]}) [2025-04-03 14:01:12.435974] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18592][T1004_CdcSrv][T1003][YB42AC140005-000631D6BFECB9ED-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.436975] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:12.437654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18592][T1004_CdcSrv][T1003][YB42AC140005-000631D6BFECB9ED-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.439055] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18592][T1004_CdcSrv][T1003][YB42AC140005-000631D6BFECB9ED-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.449447] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:407) [18592][T1004_CdcSrv][T1004][Y0-0000000000000000-0-0] [lt=0] finish to resize log external storage handler(current_ts=1743660072449443, tenant_max_cpu=4, valid_ls_count=0, other_ls_count=0, new_concurrency=0) [2025-04-03 14:01:12.456309] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.469167] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.475599] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.475641] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.475677] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.475861] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:12.476950] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.480694] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.499116] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56286 [2025-04-03 14:01:12.505173] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.505315] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.508114] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.508192] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.508223] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.508249] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.515700] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.534929] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA685-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.534950] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA685-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.534957] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA685-0-0] [lt=6] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:12.534977] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA685-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072534976, replica_locations:[]}) [2025-04-03 14:01:12.535005] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA685-0-0] [lt=26][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.535014] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA685-0-0] [lt=8][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072532151}) [2025-04-03 14:01:12.535023] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA685-0-0] [lt=8][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.535037] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA685-0-0] [lt=13][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.535145] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56288 [2025-04-03 14:01:12.539544] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660072539066]}) [2025-04-03 14:01:12.540179] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660072539577]}) [2025-04-03 14:01:12.541073] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA686-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.541091] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA686-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.541107] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA686-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072541106, replica_locations:[]}) [2025-04-03 14:01:12.541124] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA686-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.541132] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA686-0-0] [lt=7][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072539014}) [2025-04-03 14:01:12.541139] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA686-0-0] [lt=6][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.541143] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA686-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.541969] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660072540207]}) [2025-04-03 14:01:12.544457] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=2] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.556423] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.562005] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.569110] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [17832][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=1549] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.569232] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.575688] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.575743] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.575762] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.577073] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.591606] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [18288][T1003_Occam][T1003][YB42AC140005-000631D6BF9CDA67-0-0] [lt=21578] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-04-03 14:01:12.594273] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.605253] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.605303] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.605322] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.608265] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.608941] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [18032][T1001_Flush][T1001][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.609194] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:12.610430] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.610479] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18235][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.615777] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.626937] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17558][RpcIO][T0][Y0-0000000000000000-0-0] [lt=35] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.628075] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18586][T1004_DupTbLeas][T1003][YB42AC140005-000631D6C44CB927-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.630046] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:12.630473] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18586][T1004_DupTbLeas][T1003][YB42AC140005-000631D6C44CB927-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.633619] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:12.637482] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=47] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.646032] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=5][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.646389] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660072645954]}) [2025-04-03 14:01:12.647306] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.647322] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.647328] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=5] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:12.647345] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072647345, replica_locations:[]}) [2025-04-03 14:01:12.647364] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.647375] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=11][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072645599}) [2025-04-03 14:01:12.647382] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=6][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.647387] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA687-0-0] [lt=5][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.648194] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.648908] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.649072] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.660787] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.664326] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.664891] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4806991413, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:12.669354] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.675778] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.680614] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.681990] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.685351] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.703794] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56304 [2025-04-03 14:01:12.705014] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.705447] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=39][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.705494] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.705518] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.705535] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.708750] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=65] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:12.715865] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.732290] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA688-0-0] [lt=5][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.733705] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.734436] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA688-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.734458] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA688-0-0] [lt=21][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.734477] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA688-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072734476, replica_locations:[]}) [2025-04-03 14:01:12.734495] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA688-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.734503] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA688-0-0] [lt=7][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072731582}) [2025-04-03 14:01:12.734510] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA688-0-0] [lt=7][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.734515] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA688-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.753246] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA689-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.753263] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA689-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.753285] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA689-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072753284, replica_locations:[]}) [2025-04-03 14:01:12.753302] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA689-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.753309] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA689-0-0] [lt=7][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072750886}) [2025-04-03 14:01:12.753316] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA689-0-0] [lt=6][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.753320] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA689-0-0] [lt=3][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.756198] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:33444, tid:18428}]) [2025-04-03 14:01:12.756595] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.762379] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4807088252, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:12.762429] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.769448] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.775858] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.775923] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.775957] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.777463] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=72] [PALF_DUMP](palf_id=1, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:30010310, max_lsn:13483085678, max_scn:{val:1743655277903270387, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:13483027323}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:13441142638}}, last_submit_lsn:{lsn:13483026786}, last_submit_end_lsn:{lsn:13483027323}, last_submit_log_id:30010185, last_submit_log_pid:159, max_flushed_lsn:{lsn:13483026786}, max_flushed_end_lsn:{lsn:13483027323}, max_flushed_log_pid:159, committed_end_lsn:{lsn:13483027323}, last_slide_log_id:30010185, last_slide_scn:{val:1743655265084940962, v:0}, last_slide_lsn:{lsn:13483026786}, last_slide_end_lsn:{lsn:13483027323}, last_slide_log_pid:159, last_slide_log_accum_checksum:1426087181, last_fetch_end_lsn:{lsn:13524970363}, last_fetch_max_log_id:30012233, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660072681978, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7398e68530}, [StateMgr]={this:0x2b7398e6ca30, self:"172.20.0.5:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:159}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:30010186, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2b7398e732f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648235302292, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:159, prev:{version:1, config:{config_version:{proposal_id:159, config_seq:24}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648235302292, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:159, config_seq:25}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648235302292, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:159, prev_lsn:{lsn:13442084080}, prev_mode_pid:159}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:159, config_seq:25}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660072276946, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660072276946, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b7398e69c70}, [ModeMgr]={palf_id:1, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:159, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:159, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:159, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:13286744064}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:159}, log_config_meta:{version:3, proposal_id:159, prev:{version:1, config:{config_version:{proposal_id:159, config_seq:24}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648235302292, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:159, config_seq:25}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648235302292, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:159, prev_lsn:{lsn:13442084080}, prev_mode_pid:159}, log_snapshot_meta:{version:1, base_lsn:{lsn:13286744064}, prev_log_info:{log_id:29587622, lsn:{lsn:13286739795}, scn:{val:1743609917851655855, v:0}, log_proposal_id:159, accum_checksum:3749821322}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:159, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:24576}, readable_log_tail:{lsn:24576}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:13488058368}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1/1/meta", dir_fd:81, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67080192, block_header_serialize_buf_:0x2b7398e6ebb8, flashback_version:0}, log_storage:{log_tail:{lsn:13483027323}, readable_log_tail:{lsn:13483027323}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:13420953600}, min_scn:{val:1743644078510680054, v:0}, curr_block_id:200, palf_id:1}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1/1/log", dir_fd:82, min_block_id:198, max_block_id:201, curr_writable_block_id:200}, logical_block_size_:67104768, curr_block_writable_size_:5031045, block_header_serialize_buf_:0x2b7398e70838, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:-1, this:0x2b7398e6d170}, [Reconfirm]={palf_id:1, self:"172.20.0.5:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b7398e6cbb0}) [2025-04-03 14:01:12.777946] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=474][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:12.777978] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30] [PALF_DUMP](palf_id=1, self="172.20.0.5:2882", ack_info_list=[]) [2025-04-03 14:01:12.778003] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:12.778017] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4308) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13] current log_task status(palf_id=1, self="172.20.0.5:2882", log_id=30010186, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-04-03 14:01:12.778102] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=80][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.779318] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4675193372, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:12.779370] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.782277] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:12.782321] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=36] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56310 [2025-04-03 14:01:12.784129] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=28] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56310 [2025-04-03 14:01:12.784163] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=28] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:12.786914] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=10] PNIO [ratelimit] time: 1743660072786911, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007586, add_bytes: 0 [2025-04-03 14:01:12.804707] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.805510] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.805530] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.805538] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.805547] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.812107] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:87) [17994][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.815925] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.816420] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE40-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.833400] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56312 [2025-04-03 14:01:12.836714] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:12.846202] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:261) [17837][T1_ArcSender][T1][YB42AC140005-000631D6BB2CB54C-0-0] [lt=45][errcode=-4018] try free send task failed(ret=-4018) [2025-04-03 14:01:12.846895] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [17564][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=3] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.850059] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.850770] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.851063] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:12.851176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.852087] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=45] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:12.852713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.852826] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.853124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.853438] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1002}, delta=4807785167, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:12.855424] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660072854861]}) [2025-04-03 14:01:12.855716] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660072855449]}) [2025-04-03 14:01:12.856660] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.856843] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68A-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.856859] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68A-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.856865] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68A-0-0] [lt=5] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:12.856883] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68A-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072856882, replica_locations:[]}) [2025-04-03 14:01:12.856908] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68A-0-0] [lt=23][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.856916] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68A-0-0] [lt=7][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072854842}) [2025-04-03 14:01:12.856922] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68A-0-0] [lt=6][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.856927] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68A-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.862185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.862215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.862599] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.862622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.862904] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.863110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.863246] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.863382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.864361] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.864532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.869523] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.875944] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.875992] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.876008] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.877547] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.879348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879427] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879437] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879502] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879599] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879752] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879823] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.879861] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.885158] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=1] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=0) [2025-04-03 14:01:12.897614] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:12.897662] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=42] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56316 [2025-04-03 14:01:12.897725] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56316 [2025-04-03 14:01:12.897764] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=36] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:12.905554] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.905584] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.905595] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.905609] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.909713] INFO [ARCHIVE] get_send_task_ (ob_archive_sender.cpp:364) [18635][T1004_ArcSender][T1004][YB42AC140005-000631D6C0CCB54C-0-0] [lt=1] task not exist, wait time(time_cost=100082) [2025-04-03 14:01:12.916009] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=17] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"172.20.0.5:2882", switch_to_leader_ts:-1}) [2025-04-03 14:01:12.916065] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=54][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:12.925730] INFO [ARCHIVE] get_send_task_ (ob_archive_sender.cpp:364) [18639][T1004_ArcSender][T1004][YB42AC140005-000631D6C0ECB54C-0-0] [lt=23] task not exist, wait time(time_cost=100068) [2025-04-03 14:01:12.944867] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4597520568, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:12.944959] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=85] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.945138] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=44] get wrs ts(ls_id={id:1001}, delta=4795588625, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:12.945187] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.945290] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1002}, delta=4807876940, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:12.945319] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.945425] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1003}, delta=4593461935, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:12.945458] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.956645] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=40][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4529, tid:17463}]) [2025-04-03 14:01:12.956774] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.958831] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:12.959318] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.959358] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.959375] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660072958501], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.959416] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=40][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660072958501], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.959463] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660072958501]}) [2025-04-03 14:01:12.960163] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.960194] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.960219] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660072959497], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.960244] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660072959497], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.960261] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660072959497]}) [2025-04-03 14:01:12.960282] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=3][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.960332] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=48][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.960355] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=18][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.960389] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=32][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.4:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.961147] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.961175] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.961187] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=10][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.961205] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=17][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.3:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.961243] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.961273] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.961284] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660072960287], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.961302] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660072960287], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.961330] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660072960287]}) [2025-04-03 14:01:12.961414] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:12.961975] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.962019] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=43][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.962051] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=29][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.962089] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=36][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.5:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.962189] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.962221] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.962234] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660072961371], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.962228] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4807288314, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:12.962261] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660072961371], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.962290] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660072961371]}) [2025-04-03 14:01:12.962278] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.962458] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.962499] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=38][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:12.962538] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=37] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:12.962566] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:12.962598] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=19] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660072962596, replica_locations:[]}) [2025-04-03 14:01:12.962657] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=55][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:12.962674] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=16][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660072959433}) [2025-04-03 14:01:12.962693] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=17][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:12.962704] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68B-0-0] [lt=10][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:12.963405] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.963443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.963463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.963582] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.963600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.963661] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.963712] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=49][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.963729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.963729] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660072962664], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.963749] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660072962664], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:12.963743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.963771] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660072962664]}) [2025-04-03 14:01:12.963966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.963985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964460] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.964987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965255] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965416] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965802] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.965953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.966071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.966087] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.966260] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:12.968860] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}]) [2025-04-03 14:01:12.969595] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.972661] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:12.973420] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:12.976045] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660068898113, last_first_register_time_us=1743648235573699, register_time_us=1743660068898113, curr_time_us=1743660072976042) [2025-04-03 14:01:12.976567] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.976598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.976614] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.977569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.977591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.977622] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.978529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.978549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.979440] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4675393174, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:12.979479] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:12.979644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.979674] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.980168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.980196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.980528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.980552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.980846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.980866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.981164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.981178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.981418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.981438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.981696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.981729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.982013] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.982047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.982311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.982334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.982573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.982594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.982849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.982869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.983135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.983163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.983414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.983431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.983691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.983709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.983993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984049] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.984065] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] oops, alloc failed, tenant_id=1002, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=482500608, tenant_limit=536870912 [2025-04-03 14:01:12.984083] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] allocate memory fail(attr=tenant_id=1002, label=OB_LOG_REGISTER_PARENT_REQ, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.984104] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:12.984118] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] LogRpc post_request failed(ret=-4013, palf_id_=1, req={child:{server:"172.20.0.5:2882", region:"default_region", register_time_us:1743660072976091, keepalive_ts:-1}, is_to_leader:true}, server="172.20.0.3:2882") [2025-04-03 14:01:12.984149] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.20.0.5:2882") [2025-04-03 14:01:12.984185] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.984205] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.984228] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:12.984413] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:12.984436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.984907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985285] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985950] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.985957] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:12.986217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986383] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986442] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.986911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.987096] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:12.988524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.988538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.988641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.988652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.988878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.988899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989259] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.989921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990244] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990313] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.990625] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:12.990635] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9][errcode=-4013] oops, alloc failed, tenant_id=1001, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=397373440, tenant_limit=536870912 [2025-04-03 14:01:12.990644] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:12.990651] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7][errcode=-4013] rpc memory pool alloc memory failed(sz=1305, alloc_sz=16256) [2025-04-03 14:01:12.990658] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1121) [2025-04-03 14:01:12.990664] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:12.990674] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=29999841, timeout_timestamp=1743660102988242) [2025-04-03 14:01:12.990682] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:12.990689] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:12.990694] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:12.990699] WDIAG [SQL] open (ob_result_set.cpp:158) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:12.990706] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:12.990716] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=9][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:12.990725] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:12.990734] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:12.990757] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=16][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:12.990763] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:12.990767] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:12.990997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991251] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991974] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.991982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992398] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992557] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.992847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993402] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993614] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993725] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.993957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994069] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994468] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994653] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994750] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.994945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995424] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995904] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.995999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996833] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.996929] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997013] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997465] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997646] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:12.997671] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:12.997679] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=7][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:12.997691] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=11][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:12.997697] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:12.997703] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:12.997709] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:12.997715] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] query failed(ret=-4013, conn=0x2b73ba1ea050, start=1743660072988243, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:12.997722] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=6][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:12.997728] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:12.997771] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:12.997776] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] fail to get ls infos by tenant(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, inner_table_only=false) [2025-04-03 14:01:12.997782] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] fail to open iterator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.997786] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=4][errcode=-4013] ls table iterator next failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.997799] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=10][errcode=-4013] build replica map from ls table failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", mode=0) [2025-04-03 14:01:12.997816] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:188) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] check ls table failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", mode=0) [2025-04-03 14:01:12.997821] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D4-0-0] [lt=5][errcode=-4013] fail to check ls meta table(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:12.999194] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.000476] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.000490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.000497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.000631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.000638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.000797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.000803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001035] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001246] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.001900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.002263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.002270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.002533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.002539] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.002706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.002712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.002884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.002890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003465] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003515] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.003690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.003925] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.004152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.004158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.004613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.004621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.005006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.005013] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.005260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.005267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.005631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.005637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.005634] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.005668] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.005693] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.005727] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.006070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.006082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.006312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.006320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.006527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.006538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.007898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.007908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.008122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.008137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.008323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.008331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.008579] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.008587] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.008795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.008803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.009058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.009067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.009300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.009310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.010210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.010220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.010735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.010743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.010776] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.010784] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7][errcode=-4013] oops, alloc failed, tenant_id=1001, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=397373440, tenant_limit=536870912 [2025-04-03 14:01:13.010793] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:13.010814] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=21][errcode=-4013] rpc memory pool alloc memory failed(sz=1330, alloc_sz=16256) [2025-04-03 14:01:13.010820] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5][errcode=-4013] alloc buffer fail(payload_sz=1146) [2025-04-03 14:01:13.010834] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=13][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:13.010843] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=3][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=1994691, timeout_timestamp=1743660074999107) [2025-04-03 14:01:13.010851] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:13.010861] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:13.010866] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:13.010870] WDIAG [SQL] open (ob_result_set.cpp:158) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=4][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:13.010875] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=4][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:13.010879] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=3][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}) [2025-04-03 14:01:13.010886] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:13.010893] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:13.010913] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=13][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:13.010918] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:13.010922] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=3][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:13.011200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=4] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.011975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012259] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012662] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.012921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.013433] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.013440] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.013648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.013656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.013940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.013945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.014101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.014107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.014267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.014272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.014454] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.014531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.014537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.014876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.014887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015102] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015230] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.015376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015731] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.015981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.016075] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=11] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"172.20.0.5:2882", switch_to_leader_ts:-1}) [2025-04-03 14:01:13.016103] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.016318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.016329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.016621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.016631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.016752] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.016759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.016981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.016991] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017403] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.017960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018057] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018065] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018148] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018402] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018558] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.018870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019174] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.019876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.020081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.020352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.020575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.020745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.020838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.021031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.021216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.021835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.022045] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.022071] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:13.022079] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=6][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:13.022090] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1547-0-0] [lt=10][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name"}, aret=-4013, ret=-4013) [2025-04-03 14:01:13.022095] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=4][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2025-04-03 14:01:13.022101] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:13.022105] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=4][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:13.022111] WDIAG [COMMON.MYSQLP] read_without_check_sys_variable (ob_sql_client_decorator.cpp:119) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] failed to read without check sys variable(ret=-4013, sql="SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name", tenant_id=1001, check_sys_variable=false, snapshot_timestamp=-1) [2025-04-03 14:01:13.022120] WDIAG [SHARE] load (ob_core_table_proxy.cpp:436) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, sql=SELECT row_id, column_name, column_value FROM __all_core_table WHERE table_name = '__all_global_stat' ORDER BY row_id, column_name) [2025-04-03 14:01:13.022163] WDIAG [SHARE] load (ob_core_table_proxy.cpp:368) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] load failed(ret=-4013, for_update=false) [2025-04-03 14:01:13.022169] WDIAG [SHARE] get (ob_global_stat_proxy.cpp:442) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] core_table load failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:13.022174] WDIAG [SHARE] get_core_schema_version (ob_global_stat_proxy.cpp:390) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=4][errcode=-4013] get failed(ret=-4013) [2025-04-03 14:01:13.022179] WDIAG [SHARE.SCHEMA] get_core_version (ob_schema_service_sql_impl.cpp:857) [18029][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=4][errcode=-4013] get_core_schema_version failed(ret=-4013) [2025-04-03 14:01:13.026805] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.033890] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.033941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=46] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.033966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.034323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.034355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.034502] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.034519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035439] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.035947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036136] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036356] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036539] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036731] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.036930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.037199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.037295] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.037642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.037779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.037940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.038272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.038432] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.038608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.039040] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.039256] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.040708] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.041560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.042004] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.042247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.042506] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.043997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.044501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.044523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.044666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.046272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.046284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.046956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.048057] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.049568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.049995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.050472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.050825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.050974] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1001}, delta=4795694850, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:13.051635] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.052324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.052629] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.052749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.053100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.053266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.056860] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=83][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:17618, tid:17422}]) [2025-04-03 14:01:13.056860] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=66][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.062317] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=45][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.062335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.062346] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.062438] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.062452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.062458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.062556] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.062573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.062671] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.062754] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.062770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.062857] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.062949] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063038] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063123] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.063213] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063306] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063386] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063494] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.063571] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063696] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063824] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.063938] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.064065] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:13.065099] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.065695] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073065139]}) [2025-04-03 14:01:13.066070] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:13.066110] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=35][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:13.066702] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:13.066727] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=22][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:13.066913] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660073066449]}) [2025-04-03 14:01:13.067399] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=1][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:13.067425] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=23][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:13.067449] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=22][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:13.067766] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=25][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:13.067825] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=57][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:13.067846] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.4:2882") [2025-04-03 14:01:13.067861] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.3:2882") [2025-04-03 14:01:13.067876] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.5:2882") [2025-04-03 14:01:13.067905] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.067931] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=24][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.067959] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=0] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:13.067987] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=16] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073067985, replica_locations:[]}) [2025-04-03 14:01:13.068059] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=68][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.068088] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=28][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073065122}) [2025-04-03 14:01:13.068105] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=16][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.068126] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68C-0-0] [lt=19][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.069661] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=60][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.076105] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.076150] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.076200] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.077684] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.079483] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.079495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.079582] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.079626] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.079632] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.079772] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.079864] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.079901] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.079947] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.080571] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.080634] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.080642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.080684] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.080733] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.080798] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.080856] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.080992] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.081003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.081069] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.081114] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:13.081231] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-04-03 14:01:13.082588] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.082610] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28302 [2025-04-03 14:01:13.082649] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28302 [2025-04-03 14:01:13.088830] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:686) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] Cache replace map node details(ret=0, replace_node_count=0, replace_time=3533, replace_start_pos=849339, replace_num=31457) [2025-04-03 14:01:13.088873] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=10) [2025-04-03 14:01:13.105718] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660069097322, last_first_register_time_us=1743648246712872, register_time_us=1743660069097322, curr_time_us=1743660073105716) [2025-04-03 14:01:13.105800] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=33][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:13.105831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.105841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.105910] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:13.105934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.105941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.106004] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:13.106019] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.106043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.106244] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:13.106263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.106271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.106337] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:13.106417] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:13.106431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.106624] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:13.106701] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:13.106723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.106930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.107670] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.107696] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.107713] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.107725] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.110849] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=80] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:13.116164] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.144872] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4597720388, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:13.144888] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.145071] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=4795789545, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:13.145085] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.145126] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1002}, delta=4808076801, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:13.145136] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.145171] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1003}, delta=4593661796, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:13.145178] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.154664] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:110) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=71] try gc retain ctx [2025-04-03 14:01:13.156917] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=36][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.162385] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4807488753, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:13.162433] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.168467] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.168495] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660073167836]}) [2025-04-03 14:01:13.168505] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:171) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", gts_tenant_info={v:1}) [2025-04-03 14:01:13.168743] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073168516]}) [2025-04-03 14:01:13.168904] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660073168751]}) [2025-04-03 14:01:13.169146] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660073168913]}) [2025-04-03 14:01:13.169628] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.169647] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.169660] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.169671] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=6] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:13.169658] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660073169333]}) [2025-04-03 14:01:13.169687] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073169686, replica_locations:[]}) [2025-04-03 14:01:13.169707] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=18][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.169714] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=6][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073168484}) [2025-04-03 14:01:13.169721] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=6][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.169725] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68D-0-0] [lt=3][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.169764] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.172856] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.172874] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.172880] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=5] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.172888] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=5] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:13.172896] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=4] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073172895, replica_locations:[]}) [2025-04-03 14:01:13.172913] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.172921] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=7][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073171674}) [2025-04-03 14:01:13.172930] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=8][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.172934] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68E-0-0] [lt=3][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.176192] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.176244] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.176280] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.177745] INFO [PALF] log_loop_ (log_loop_thread.cpp:175) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] LogLoopThread round_cost_time(us)(round_cost_time=26) [2025-04-03 14:01:13.177762] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.180041] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4675594532, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:13.180118] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=71] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.190383] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [17917][T1001_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=1511] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):512, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):184, sequence:1}) [2025-04-03 14:01:13.195727] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.195784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=52] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.195803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.196045] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.196072] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.196880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.196925] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=42] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.197209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.197234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.197480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.197708] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.198145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.198762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.199115] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.199330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.199461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.199625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.200090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.200337] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.200585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.200797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.201139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.201434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.201773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.202194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.202701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.202864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.203262] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.203679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.204343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.204870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.205826] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.205884] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=46] [PALF_DUMP](palf_id=1002, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1002, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:608085, max_lsn:74025692, max_scn:{val:1743655277950177453, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:74011080}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:32082652}}, last_submit_lsn:{lsn:74010958}, last_submit_end_lsn:{lsn:74011080}, last_submit_log_id:607965, last_submit_log_pid:1, max_flushed_lsn:{lsn:74010958}, max_flushed_end_lsn:{lsn:74011080}, max_flushed_log_pid:1, committed_end_lsn:{lsn:74011080}, last_slide_log_id:607965, last_slide_scn:{val:1743655265068231480, v:0}, last_slide_lsn:{lsn:74010958}, last_slide_end_lsn:{lsn:74011080}, last_slide_log_pid:1, last_slide_log_accum_checksum:1277512058, last_fetch_end_lsn:{lsn:115954120}, last_fetch_max_log_id:610013, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660072508181, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b73ab3c0530}, [StateMgr]={this:0x2b73ab3c4a30, self:"172.20.0.5:2882", palf_id:1002, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:607966, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2b73ab3cb2f0, allow_vote:true}, [ConfigMgr]={palf_id:1002, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246130462, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:1, config_seq:26}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246130462, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:26, lock_time:1743614588772911}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:27}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246130462, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:26, lock_time:1743614588772911}}, prev_log_proposal_id:1, prev_lsn:{lsn:66292490}, prev_mode_pid:1}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1, config_seq:27}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660071204208, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660071204208, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b73ab3c1c70}, [ModeMgr]={palf_id:1002, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1743590034342901212, v:0}}, accepted_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1743590034342901212, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1743590034342901212, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1002, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1}, log_config_meta:{version:3, proposal_id:1, prev:{version:1, config:{config_version:{proposal_id:1, config_seq:26}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246130462, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:26, lock_time:1743614588772911}}, curr:{version:1, config:{config_version:{proposal_id:1, config_seq:27}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246130462, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:26, lock_time:1743614588772911}}, prev_log_proposal_id:1, prev_lsn:{lsn:66292490}, prev_mode_pid:1}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:0, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1, mode_version:0, access_mode:1, ref_scn:{val:1743590034342901212, v:0}}}, log_meta_storage:{log_tail:{lsn:20480}, readable_log_tail:{lsn:20480}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1002}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1002/meta", dir_fd:122, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67084288, block_header_serialize_buf_:0x2b73ab3c6bb8, flashback_version:0}, log_storage:{log_tail:{lsn:74011080}, readable_log_tail:{lsn:74011080}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1743648992737306959, v:0}, curr_block_id:1, palf_id:1002}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1002/log", dir_fd:123, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:60198456, block_header_serialize_buf_:0x2b73ab3c8838, flashback_version:0}, palf_epoch:7, last_purge_throttling_ts:-1, this:0x2b73ab3c5170}, [Reconfirm]={palf_id:1002, self:"172.20.0.5:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b73ab3c4bb0}) [2025-04-03 14:01:13.206375] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=483][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:13.206399] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23] [PALF_DUMP](palf_id=1002, self="172.20.0.5:2882", ack_info_list=[]) [2025-04-03 14:01:13.206415] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1002, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:13.206432] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.206451] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.206475] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.206733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.206970] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F8-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.209712] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D9-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.210314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.211496] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D9-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.213084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D9-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.213471] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D9-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.213497] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2D9-0-0] [lt=25][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.216252] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.225081] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.237317] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18111][T1002_LogUpdate][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.238761] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68F-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.238783] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68F-0-0] [lt=21][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.238803] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68F-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073238803, replica_locations:[]}) [2025-04-03 14:01:13.238837] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68F-0-0] [lt=31][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.238854] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68F-0-0] [lt=16][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073234623}) [2025-04-03 14:01:13.238862] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68F-0-0] [lt=7][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.238867] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA68F-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.243995] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE373-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.257050] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.265683] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28308 [2025-04-03 14:01:13.269817] INFO [PALF] log_loop_ (log_loop_thread.cpp:175) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=61] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.270517] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.270548] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.270972] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073270524]}) [2025-04-03 14:01:13.271126] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0] ====== tenant freeze timer task ====== [2025-04-03 14:01:13.271178] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1555) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=39] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.271302] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA690-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.271317] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA690-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.271331] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA690-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073271330, replica_locations:[]}) [2025-04-03 14:01:13.271350] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA690-0-0] [lt=18][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.271367] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA690-0-0] [lt=16][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073270507}) [2025-04-03 14:01:13.271374] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA690-0-0] [lt=6][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.271378] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA690-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.271582] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660073270992]}) [2025-04-03 14:01:13.276260] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.276282] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.276291] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.277846] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.279701] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.280350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.282785] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28312 [2025-04-03 14:01:13.286881] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28314 [2025-04-03 14:01:13.288977] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=9) [2025-04-03 14:01:13.299519] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.299556] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28316 [2025-04-03 14:01:13.299613] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28316 [2025-04-03 14:01:13.299630] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.305898] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.305950] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.305988] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.306005] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.310572] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [18034][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-04-03 14:01:13.310601] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18034][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=18] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-04-03 14:01:13.310612] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [18034][T1001_ObTimer][T1001][Y0-0000000000000000-0-0] [lt=8] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=2374) [2025-04-03 14:01:13.316374] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.345519] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4597921596, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:13.345576] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=49] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.346056] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=55] get wrs ts(ls_id={id:1001}, delta=4795989653, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:13.346095] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.346557] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1002}, delta=4808278001, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:13.346625] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=62] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.346762] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1003}, delta=4593862996, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:13.346791] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.357124] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.362566] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4807688597, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:13.362619] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.364021] INFO [ARCHIVE] get_send_task_ (ob_archive_sender.cpp:364) [18250][T1002_ArcSender][T1002][YB42AC140005-000631D6BE4CB54C-0-0] [lt=1533] task not exist, wait time(time_cost=100057) [2025-04-03 14:01:13.369982] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.374467] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660073373389]}) [2025-04-03 14:01:13.375763] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073374503]}) [2025-04-03 14:01:13.376331] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.376373] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.376387] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.376951] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660073375797]}) [2025-04-03 14:01:13.377730] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.377754] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=23][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.377763] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.377790] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=20] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:13.377819] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073377806, replica_locations:[]}) [2025-04-03 14:01:13.377856] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=35][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.377869] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=12][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073374449}) [2025-04-03 14:01:13.377879] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=9][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.377885] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA691-0-0] [lt=5][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.377930] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660073376969]}) [2025-04-03 14:01:13.377922] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.378754] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660073378178]}) [2025-04-03 14:01:13.379784] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4675794357, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:13.379828] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.398675] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28318 [2025-04-03 14:01:13.402235] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.405981] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.406045] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=58][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.406083] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.406102] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.414672] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:214) [18603][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.414794] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:214) [18603][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0] start to check and clear obj lock(ls->get_ls_id()={id:1002}) [2025-04-03 14:01:13.416456] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=53][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.430075] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [18401][T1003_LockWaitM][T1003][Y0-0000000000000000-0-0] [lt=24] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.449131] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.449185] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=33] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28322 [2025-04-03 14:01:13.449267] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28322 [2025-04-03 14:01:13.449284] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.449657] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4598025859, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:13.449696] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.449773] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1001}, delta=4796093916, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:13.449796] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.450176] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=58] get wrs ts(ls_id={id:1002}, delta=4808381172, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:13.450213] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.450453] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1003}, delta=4593966167, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:13.450474] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.457211] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=71][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.466707] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4807792714, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:13.466737] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.467965] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.468007] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28324 [2025-04-03 14:01:13.468102] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28324 [2025-04-03 14:01:13.468119] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.470193] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.473741] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.473785] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=36] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28326 [2025-04-03 14:01:13.473885] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28326 [2025-04-03 14:01:13.473930] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.476426] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.476497] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.476518] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.478004] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.480892] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.487137] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA692-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.487192] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA692-0-0] [lt=52][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.487234] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA692-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073487231, replica_locations:[]}) [2025-04-03 14:01:13.487292] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA692-0-0] [lt=53][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.487315] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA692-0-0] [lt=21][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073481399}) [2025-04-03 14:01:13.487342] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA692-0-0] [lt=25][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.487356] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA692-0-0] [lt=13][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.489094] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.499287] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=20][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:13.499319] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=30][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660075703884, now=[mts=1743660072285492], now0=[mts=1743660072285492], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:13.499347] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=24][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a4335d80, tx_id:{txid:76175009}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660072278095, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:13.499420] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=67][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73961321a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a4335d80}) [2025-04-03 14:01:13.499483] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28332 [2025-04-03 14:01:13.499468] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=46][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a4335d80, tx_id:{txid:76175009}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660072278095, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:13.499525] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=54] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.499624] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=97][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:13.499663] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=21][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:13.499675] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:13.499686] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:13.499696] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:13.499714] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=15][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=21, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:13.503209] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.506135] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.507660] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.507709] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.509014] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.509088] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.509109] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.513004] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=44] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:13.513219] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.513233] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28336 [2025-04-03 14:01:13.513268] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28336 [2025-04-03 14:01:13.513275] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.516529] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.520827] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:21, local_retry_times:21, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:13.520914] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=64][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:13.520929] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=14][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:13.520946] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:13.521876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.524759] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.524879] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:13.527068] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:13.530258] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.530291] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28338 [2025-04-03 14:01:13.530353] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28338 [2025-04-03 14:01:13.530367] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.545495] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=4598121063, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:13.545567] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=63] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.545951] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1001}, delta=4796190288, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:13.545992] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=35] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.546432] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1002}, delta=4808477544, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:13.557302] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.562798] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=4807889281, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:13.562867] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=62] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.567379] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.567421] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=32] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28340 [2025-04-03 14:01:13.567503] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28340 [2025-04-03 14:01:13.567525] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.570635] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.576506] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.576543] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.576568] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.578134] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.579975] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4675994267, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:13.580053] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.583261] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.583323] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28342 [2025-04-03 14:01:13.583399] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28342 [2025-04-03 14:01:13.583433] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.591170] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660073590266]}) [2025-04-03 14:01:13.593591] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073591238]}) [2025-04-03 14:01:13.593643] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=50] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073591238]}) [2025-04-03 14:01:13.595703] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.595754] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28344 [2025-04-03 14:01:13.595875] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28344 [2025-04-03 14:01:13.595902] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.596618] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660073593674]}) [2025-04-03 14:01:13.597841] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660073596658]}) [2025-04-03 14:01:13.599520] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660073598262]}) [2025-04-03 14:01:13.599607] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.600533] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.600576] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=40][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.600592] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.600622] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:13.600646] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073600643, replica_locations:[]}) [2025-04-03 14:01:13.600705] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=54][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.600723] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=16][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073591142}) [2025-04-03 14:01:13.600738] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=14][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.600748] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA693-0-0] [lt=9][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.602761] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=41][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:13.602803] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=39][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660072395657], now0=[mts=1743660072395657], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:13.602864] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=57][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719466}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660072394891, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:13.602968] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=97][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:13.603012] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=42][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719466}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660072394891, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:13.603092] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=76] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f7dce008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660072394892, tx_desc={this:0x2b7399219100, tx_id:{txid:47719466}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660072394891, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660054039470, use_das=false, nested_level=0, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}, plan=0x2b73976ba050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:13.603209] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=114][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:13.603254] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=41][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:13.603284] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=27][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:13.603319] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=31][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:13.603331] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=9][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:13.603349] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=16][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=15, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:13.603381] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=27] will sleep(sleep_us=15000, remain_us=10436080, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=15, v.err_=-4283, timeout_timestamp=1743660084039460) [2025-04-03 14:01:13.606100] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.606177] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=70][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.606197] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.606213] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.616604] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.618489] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=19][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:15, local_retry_times:15, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:13.618540] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.618592] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=80][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:13.618608] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:13.618627] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=17][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:13.618651] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:13.619218] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.620759] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.620830] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:13.626842] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17558][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:13.633689] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.633717] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28346 [2025-04-03 14:01:13.633775] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28346 [2025-04-03 14:01:13.633786] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.638403] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.647239] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17893][T1001_Occam][T1][YB42AC140005-000631D6BD7CDA1E-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.647462] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:13.648226] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.648247] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28348 [2025-04-03 14:01:13.648281] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28348 [2025-04-03 14:01:13.648288] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.650390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.651250] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.651403] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4598227082, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:13.651445] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.651704] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=34] get wrs ts(ls_id={id:1001}, delta=4796296214, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:13.651756] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=47] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.652217] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1002}, delta=4808583470, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:13.652261] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=39] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.652349] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1003}, delta=4594168465, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:13.652389] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.657369] INFO [PALF] log_loop_ (log_loop_thread.cpp:175) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=49] LogLoopThread round_cost_time(us)(round_cost_time=83) [2025-04-03 14:01:13.657419] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.662537] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:1044) [18186][T1002_FreInfoRe][T1002][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.662716] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=28] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1003, server_version={version:{val:1743659951195473000, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=122467238, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.20.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=56746, succ_cluster_heartbeat_count=56746, cluster_heartbeat_interval=1000000, local_cluster_version={val:1743654917194722813, v:0}, local_cluster_delta=5156467989, force_self_check=false, weak_read_refresh_interval=100000) [2025-04-03 14:01:13.662943] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=60][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.664900] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.665236] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4807991531, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:13.665264] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.666574] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:13.666618] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28350 [2025-04-03 14:01:13.666662] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28350 [2025-04-03 14:01:13.666672] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.670700] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.676597] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.676660] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.676708] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.678233] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.680881] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.680954] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4676094839, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:13.684087] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.684107] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28352 [2025-04-03 14:01:13.684144] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28352 [2025-04-03 14:01:13.684153] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.689237] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=46] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=7) [2025-04-03 14:01:13.700237] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.700257] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28354 [2025-04-03 14:01:13.700301] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28354 [2025-04-03 14:01:13.700307] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.700781] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660073700330]}) [2025-04-03 14:01:13.701682] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=22][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.701836] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073700827]}) [2025-04-03 14:01:13.702762] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660073701859]}) [2025-04-03 14:01:13.703337] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660073702780]}) [2025-04-03 14:01:13.703802] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.703844] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=41][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.703851] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=6] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.703867] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:13.703882] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=6] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073703881, replica_locations:[]}) [2025-04-03 14:01:13.703906] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=21][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.703915] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=8][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073700766}) [2025-04-03 14:01:13.703925] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=9][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.703938] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA694-0-0] [lt=13][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.704104] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660073703547]}) [2025-04-03 14:01:13.705208] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=51] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:13.706154] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.706217] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=56][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.706239] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] re_register_parent reason(palf_id=1001, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660069698124, last_first_register_time_us=1743648246815801, register_time_us=1743660069698124, curr_time_us=1743660073706237) [2025-04-03 14:01:13.707997] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1001, self="172.20.0.5:2882") [2025-04-03 14:01:13.708045] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.708066] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.714244] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.715691] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.716000] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=1] cannot_recycle_log_size statistics(cannot_recycle_log_size=129898528, threshold=483183820, need_update_checkpoint_scn=false) [2025-04-03 14:01:13.716698] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.737630] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [18303][T1003_PalfGC][T1003][Y0-0000000000000000-0-0] [lt=552] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.738149] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=6][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.739450] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.739489] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=37][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.739508] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=17] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.739572] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073739570, replica_locations:[]}) [2025-04-03 14:01:13.739620] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=45][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.739663] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=19][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073736993}) [2025-04-03 14:01:13.739681] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=16][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.739692] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA695-0-0] [lt=10][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.742919] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:214) [18222][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=781] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.743094] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA696-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.743133] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA696-0-0] [lt=37][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.743154] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA696-0-0] [lt=20] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.743188] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA696-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073743186, replica_locations:[]}) [2025-04-03 14:01:13.743238] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA696-0-0] [lt=47][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.743255] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA696-0-0] [lt=16][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073740303}) [2025-04-03 14:01:13.743432] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA696-0-0] [lt=175][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.743452] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA696-0-0] [lt=19][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.745778] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1001}, delta=4796389553, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:13.747209] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.750568] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.753219] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.757506] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.758788] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=16][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:20004, tid:18428}]) [2025-04-03 14:01:13.763212] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=4808089618, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:13.763263] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.767869] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:13.767913] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28358 [2025-04-03 14:01:13.768000] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28358 [2025-04-03 14:01:13.768038] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.770783] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.774917] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [18419][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-04-03 14:01:13.774967] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18419][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=35] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-04-03 14:01:13.774985] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [18419][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=2375) [2025-04-03 14:01:13.777007] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.777067] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.777384] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.777412] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.777667] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.777722] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.778138] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.778179] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.778347] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.778536] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.778561] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.778847] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.778871] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.779262] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.779286] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.779587] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.779615] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.779982] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.779999] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.780113] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4676194093, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:13.780150] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.780440] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.780458] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.780703] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.780717] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.781157] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.781175] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.781532] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.781548] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.781758] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.781773] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.782019] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.782049] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.782353] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.782380] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.782696] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.782711] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.783097] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:13.783116] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:13.783208] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:4768903583}, prev_lsn={lsn:4768903461}, fetch_start_log_id=30980588, last_slide_log_id=30980587, fetch_log_size=41943040, accepted_mode_pid=450, last_fetch_req_time=1743660073783207, last_fetch_end_lsn={lsn:4810846623}, last_fetch_max_log_id=30982635, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:30980587, max_lsn:4768903583, max_scn:{val:1743655263687025503, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:4768903583}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:4726983076}}, last_submit_lsn:{lsn:4768903461}, last_submit_end_lsn:{lsn:4768903583}, last_submit_log_id:30980587, last_submit_log_pid:450, max_flushed_lsn:{lsn:4768903461}, max_flushed_end_lsn:{lsn:4768903583}, max_flushed_log_pid:450, committed_end_lsn:{lsn:4768903583}, last_slide_log_id:30980587, last_slide_scn:{val:1743655263687025503, v:0}, last_slide_lsn:{lsn:4768903461}, last_slide_end_lsn:{lsn:4768903583}, last_slide_log_pid:450, last_slide_log_accum_checksum:1659985352, last_fetch_end_lsn:{lsn:4810846623}, last_fetch_max_log_id:30982635, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660073783207, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b73485f0530}) [2025-04-03 14:01:13.783317] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=97][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.783381] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.783398] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.783402] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.783416] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28360 [2025-04-03 14:01:13.783457] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28360 [2025-04-03 14:01:13.783465] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.786606] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:13.786629] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28362 [2025-04-03 14:01:13.786665] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28362 [2025-04-03 14:01:13.786672] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.787879] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=43] PNIO [ratelimit] time: 1743660073787876, bytes: 0, bw: 0.000000 MB/s, add_ts: 1000965, add_bytes: 0 [2025-04-03 14:01:13.790320] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1962) [18323][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=0] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1003) [2025-04-03 14:01:13.790356] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1464) [18323][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=30] start lib cache evict(tenant_id=1003, mem_hold=24829952, mem_limit=53687090, cache_obj_num=77, cache_node_num=77) [2025-04-03 14:01:13.790376] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1481) [18323][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=18] end lib cache evict(tenant_id=1003, cache_evict_num=0, mem_hold=24829952, mem_limit=53687090, cache_obj_num=77, cache_node_num=77) [2025-04-03 14:01:13.790393] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2694) [18323][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=14] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:13.797107] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2449) [18323][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=11] Dumping All Cache Objs(alloc_obj_list.count()=77, alloc_obj_list=[{obj_id:1, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175669}, {obj_id:3, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176543}, {obj_id:11, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191770}, {obj_id:12, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187597}, {obj_id:13, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183126}, {obj_id:16, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197324}, {obj_id:18, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189071}, {obj_id:20, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186886}, {obj_id:21, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:3, added_to_lc:true, mem_used:272122}, {obj_id:23, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173048}, {obj_id:25, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180892}, {obj_id:26, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224434}, {obj_id:37, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175597}, {obj_id:38, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176712}, {obj_id:39, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199789}, {obj_id:41, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189114}, {obj_id:43, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176578}, {obj_id:44, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181961}, {obj_id:53, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:228343}, {obj_id:54, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223949}, {obj_id:57, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178241}, {obj_id:60, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198356}, {obj_id:96, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183845}, {obj_id:98, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:387104}, {obj_id:99, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:514590}, {obj_id:100, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174057}, {obj_id:101, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178338}, {obj_id:102, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244232}, {obj_id:103, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220628}, {obj_id:104, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198182}, {obj_id:105, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188699}, {obj_id:110, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:193794}, {obj_id:112, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187623}, {obj_id:113, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243682}, {obj_id:114, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:247472}, {obj_id:115, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:232707}, {obj_id:116, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230761}, {obj_id:117, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:204521}, {obj_id:118, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179472}, {obj_id:119, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205900}, {obj_id:120, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:202842}, {obj_id:121, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187391}, {obj_id:122, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:189667}, {obj_id:123, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230530}, {obj_id:124, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:227796}, {obj_id:125, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:234881}, {obj_id:126, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:233973}, {obj_id:127, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:239235}, {obj_id:128, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:240493}, {obj_id:129, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:241444}, {obj_id:130, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:243788}, {obj_id:131, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237050}, {obj_id:132, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:237141}, {obj_id:133, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:258907}, {obj_id:134, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:271546}, {obj_id:135, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:269853}, {obj_id:136, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:287749}, {obj_id:137, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:194896}, {obj_id:138, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:306973}, {obj_id:139, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:344363}, {obj_id:140, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:444743}, {obj_id:141, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:287334}, {obj_id:142, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:552443}, {obj_id:143, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:316886}, {obj_id:144, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:435943}, {obj_id:145, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:542682}, {obj_id:146, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196673}, {obj_id:147, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:346354}, {obj_id:148, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:405282}, {obj_id:149, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181636}, {obj_id:150, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:431713}, {obj_id:151, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:535001}, {obj_id:152, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178218}, {obj_id:153, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179502}, {obj_id:154, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192895}, {obj_id:155, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178241}, {obj_id:156, tenant_id:1003, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187768}]) [2025-04-03 14:01:13.797465] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2702) [18323][T1003_PlanCache][T1003][Y0-0000000000000000-0-0] [lt=352] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:13.805142] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660073804413]}) [2025-04-03 14:01:13.805585] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.805607] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28364 [2025-04-03 14:01:13.805650] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28364 [2025-04-03 14:01:13.805662] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.806080] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073805182]}) [2025-04-03 14:01:13.806217] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.806237] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.806247] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.806257] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.806330] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660073806105]}) [2025-04-03 14:01:13.806760] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660073806352]}) [2025-04-03 14:01:13.807395] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.807594] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660073807065]}) [2025-04-03 14:01:13.808392] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.808416] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=22][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.808423] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=6] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.808520] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=8] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:13.808849] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073808847, replica_locations:[]}) [2025-04-03 14:01:13.808885] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=33][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.808895] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=9][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073805116}) [2025-04-03 14:01:13.808904] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=8][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.808909] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA697-0-0] [lt=5][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.816765] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.833862] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1962) [17719][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=0] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-04-03 14:01:13.833906] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1464) [17719][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=39] start lib cache evict(tenant_id=1, mem_hold=62914560, mem_limit=107374180, cache_obj_num=170, cache_node_num=170) [2025-04-03 14:01:13.833938] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1481) [17719][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=29] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=62914560, mem_limit=107374180, cache_obj_num=170, cache_node_num=170) [2025-04-03 14:01:13.833954] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2694) [17719][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=14] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:13.841169] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2449) [17719][T1_PlanCacheEvi][T1][Y0-0000000000000000-0-0] [lt=9] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.848042] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [17564][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=29] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.850325] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1002}, delta=4808781256, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:13.850362] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.850924] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.850969] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.851329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.851491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.851635] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=2] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28368 [2025-04-03 14:01:13.851652] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=56] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.851689] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=27] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.857585] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.863436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.863464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.863775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.863798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.864135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.864159] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.864604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.864627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.864960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.864973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.865594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.865624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.865925] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.865945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.866208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.866232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.866538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.866563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.866803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.866848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=44] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.867194] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.867217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.867234] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=33] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28370 [2025-04-03 14:01:13.867230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.867294] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28370 [2025-04-03 14:01:13.867312] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.867739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.867766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.868129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.868157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.868332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.868354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.868517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.868543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.868711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.868732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.869258] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4808196028, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:13.869314] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=48] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.870862] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=61][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.876762] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.876826] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.876844] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.878431] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.880220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.880260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.880348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.880371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.880464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.880484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881536] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.881987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882007] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882089] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882180] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882465] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882557] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882649] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.882850] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4676296943, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:13.882897] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.883885] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.883930] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28372 [2025-04-03 14:01:13.884002] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28372 [2025-04-03 14:01:13.884046] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=39] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.889386] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=43] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=6) [2025-04-03 14:01:13.892947] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:13.892984] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28374 [2025-04-03 14:01:13.893074] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28374 [2025-04-03 14:01:13.893092] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.899365] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:13.899404] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=32] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28376 [2025-04-03 14:01:13.899483] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28376 [2025-04-03 14:01:13.899507] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:13.906308] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.906363] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.906384] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.906407] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.908372] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660073908068]}) [2025-04-03 14:01:13.908867] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660073908394]}) [2025-04-03 14:01:13.909189] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660073908880]}) [2025-04-03 14:01:13.909538] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660073909201]}) [2025-04-03 14:01:13.910476] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660073909738]}) [2025-04-03 14:01:13.913699] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.913738] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=37][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:13.913752] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:13.913805] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=41] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:13.913871] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=51] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660073913868, replica_locations:[]}) [2025-04-03 14:01:13.913940] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=65][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:13.913959] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=18][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660073908357}) [2025-04-03 14:01:13.913976] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=16][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:13.913987] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA698-0-0] [lt=10][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:13.916856] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:13.919848] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=121] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:13.945975] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4598521292, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:13.946046] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=63] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.946180] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1001}, delta=4796590627, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:13.946209] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.946304] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1002}, delta=4808877883, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:13.946325] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.946428] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1003}, delta=4594462878, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:13.946449] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.957683] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.959297] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=71][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4016, tid:17533}]) [2025-04-03 14:01:13.963232] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=4808289290, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:13.963250] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:13.965943] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.966479] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=22][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.966541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.966582] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.966732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.966753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.966849] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=40][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.966912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.966933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967241] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967401] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967558] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967827] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967895] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.967977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968374] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.968417] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:13.969006] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=36] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1002}]) [2025-04-03 14:01:13.970947] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.974876] INFO [CLOG] run1 (ob_garbage_collector.cpp:1352) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] Garbage Collector is running(seq_=1185, gc_interval=10000000) [2025-04-03 14:01:13.975017] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.975047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975189] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975303] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.975981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.976070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.976263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.976411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.976458] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:13.976477] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4013] oops, alloc failed, tenant_id=1001, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=397373440, tenant_limit=536870912 [2025-04-03 14:01:13.976493] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_QUERY_LS_IS_VALID_MEMBER, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:13.976505] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] rpc memory pool alloc memory failed(sz=225, alloc_sz=16256) [2025-04-03 14:01:13.976520] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] alloc buffer fail(payload_sz=41) [2025-04-03 14:01:13.976530] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:13.976559] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1445) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] gc_check_member_list_ cost time(ret=0, time_us=1664) [2025-04-03 14:01:13.976633] INFO [STORAGE.TRANS] print_statistics (ob_xa_define.cpp:469) [18615][T1004_ObXAHbWor][T1004][Y0-0000000000000000-0-0] [lt=917] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.976833] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.976847] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.976854] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.977507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.977525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.977675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.977896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.978052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.978211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.978358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.978532] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:13.978551] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.978897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.979055] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=1, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:13.979085] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=28][errcode=-4013] rpc memory pool alloc memory failed(sz=1325, alloc_sz=16256) [2025-04-03 14:01:13.979096] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=10][errcode=-4013] alloc buffer fail(payload_sz=1141) [2025-04-03 14:01:13.979106] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=9][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:13.979119] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=7][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1, svr="172.20.0.4:2882", timeout=29999147, timeout_timestamp=1743660103976587) [2025-04-03 14:01:13.979134] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=14][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:13.979146] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=8][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:13.979156] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=8][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:13.979166] WDIAG [SQL] open (ob_result_set.cpp:158) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=8][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:13.979176] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=9][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:13.979186] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=8][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}) [2025-04-03 14:01:13.979199] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=12][errcode=-4013] execute failed(ret=-4013, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:13.979227] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=25][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:13.979274] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=37][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:13.979295] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=9][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:13.979304] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=8][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:13.979321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [18012][T1001_GCCollect][T1][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=8][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:13.979333] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [18012][T1001_GCCollect][T1001][YB42AC140005-000631D6C3ECBCA8-0-0] [lt=11][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, aret=-4013, ret=-4013) [2025-04-03 14:01:13.979343] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-04-03 14:01:13.979355] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1) [2025-04-03 14:01:13.979366] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1) [2025-04-03 14:01:13.979375] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] query failed(ret=-4013, conn=0x2b739baea050, start=1743660073977319, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-04-03 14:01:13.979389] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:13.979398] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:940) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] failed to read(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-04-03 14:01:13.979484] WDIAG [SHARE] get_ls_status_ (ob_ls_status_operator.cpp:1020) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] fail to inner get ls status info(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001, tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-04-03 14:01:13.979500] WDIAG [SHARE] get_ls_status_info (ob_ls_status_operator.cpp:637) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] failed to get ls status(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", id={id:1}, tenant_id=1001) [2025-04-03 14:01:13.979513] INFO [CLOG] get_ls_status_from_table (ob_garbage_collector.cpp:1384) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] failed to get ls status info from table(ret=-4013, tenant_id=1001, ls_id={id:1}) [2025-04-03 14:01:13.979527] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1630) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] failed to get ls status from table(ret=-4013, tenant_id=1001, ls_id={id:1}) [2025-04-03 14:01:13.979542] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1717) [18012][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=0] execute_gc cost time(ret=0, time_us=0) [2025-04-03 14:01:13.979978] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.979865] INFO [STORAGE] get_kept_multi_version_start (ob_tablet.cpp:4081) [17842][T1_SSTableGC][T1][YB42AC140005-000631D6C54CB6D7-0-0] [lt=17] tablet multi version start not advance for a long time(ret=0, ls_id={id:1}, tablet_id={id:60476}, multi_version_start=1743616868967014347, old_min_reserved_snapshot=1743653456724908819, min_medium_snapshot=9223372036854775807, ls_min_reserved_snapshot=1743653456724908819, old_tablet_multi_version_start=1743616814501038717, old_tablet_snaphsot_version=1743616868967014347, tablet={this:0x2b7309706bc0, is_inited:true, wash_score:1743653587454265640, hold_ref_cnt:true, ref_cnt:2, version:2, length:0, tablet_addr:[1732007-1344-0-2305843666344484864-0](offset=794624,size=612,type=2,seq=0), allocator:null, tablet_meta:{version:1, ls_id:{id:1}, tablet_id:{id:60476}, data_tablet_id:{id:476}, ref_tablet_id:{id:0}, has_next_tablet:false, create_scn:{val:18446744073709551615, v:3}, start_scn:{val:1, v:0}, clog_checkpoint_scn:{val:1, v:0}, ddl_checkpoint_scn:{val:1, v:0}, snapshot_version:1743616868967014347, multi_version_start:1743616814501038717, compat_mode:0, ha_status:{restore_status:0, data_status:0, expected_status:0, reserved:0}, report_status:{merge_snapshot_version:1743616801106230654, cur_report_version:1743616801106230654, data_checksum:0, row_count:0}, table_store_flag:{with_major_sstable:1}, ddl_start_scn:{val:0, v:0}, ddl_snapshot_version:0, max_sync_storage_schema_version:0, max_serialized_medium_scn:0, ddl_execution_id:-1, ddl_data_format_version:0, ddl_commit_scn:{val:0, v:0}, mds_checkpoint_scn:{val:1743618797082332550, v:0}, transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:0, v:0}, transfer_seq:0, has_transfer_table:false}, create_schema_version:0, space_usage:{shared_data_size:0, data_size:0, shared_meta_size:0, meta_size:0, occupy_bytes:0}}, table_store_addr:{ptr:{this:0x2b730970727c, version:257, major_tables:{cnt:1, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:true, sstable_arrays:{major_tables:{{i:0, addr:[1732007-1344-0-2305843315230900224-0](offset=786432,size=285,type=2,seq=0), type:"MAJOR", tablet_id:{id:60476}, scn_range:{start_scn:{val:0, v:0}, end_scn:{val:1743616801106230654, v:0}}, snapshot_version:1743616801106230654, max_merge_version:1743616801106230654}}, minor_tables:{null}, ddl_sstables:{null}, ddl_mem_sstables:{null}, meta_major_tables:{null}, memtables_:{null}}}, addr:[1732007-1344-0-2305843251880136704-0](offset=790528,size=226,type=2,seq=0)}, storage_schema_addr:{ptr:null, addr:[1732007-1344-0-2305843163833307362-0](offset=790754,size=144,type=2,seq=0)}, mds_data:{is_inited:true, tablet_status:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[1732007-1344-0-2305843251880137074-0](offset=790898,size=226,type=2,seq=0)}}, aux_tablet_info:{uncommitted_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, committed_kv:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}}, extra_medium_info:{info:4097, compat:1, last_compaction_type:0, wait_check_flag:1, last_medium_scn:1743616801106230654}, medium_info_list:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, auto_inc_seq:{ptr:(nil), addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0)}, tablet_status_cache:{tablet_status:{val:1, str:"NORMAL"}, transfer_scn:{val:18446744073709551615, v:3}, transfer_ls_id:{id:-1}, data_type:1, create_commit_scn:{val:1737450196599508192, v:0}, create_commit_version:1737450196599508192, delete_commit_scn:{val:18446744073709551615, v:3}, delete_commit_version:-1, transfer_out_commit_version:-1}}, ddl_kvs:null, ddl_kv_count:0, memtables:[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], memtable_count:0}) [2025-04-03 14:01:13.989889] INFO [STORAGE] runTimerTask (ob_tenant_tablet_scheduler.cpp:216) [17842][T1_SSTableGC][T1][YB42AC140005-000631D6C54CB6D7-0-0] [lt=179] SSTableGCTask(cost_ts=10918) [2025-04-03 14:01:13.991491] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.994313] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=49][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:13.998065] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=4][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:13.998090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998315] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.998837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999301] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999599] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:13.999921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.000016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.000033] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.001279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.001298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.001485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.001499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.001687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.001707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.001904] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.001916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.002076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.002088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.002227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.002331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.002483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.002753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.002907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.003084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.003466] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.003481] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=14][errcode=-4013] oops, alloc failed, tenant_id=1001, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=397373440, tenant_limit=536870912 [2025-04-03 14:01:14.003495] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=9][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.003506] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=10][errcode=-4013] rpc memory pool alloc memory failed(sz=1305, alloc_sz=16256) [2025-04-03 14:01:14.003514] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=8][errcode=-4013] alloc buffer fail(payload_sz=1121) [2025-04-03 14:01:14.003523] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=8][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:14.003536] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=29999876, timeout_timestamp=1743660104001076) [2025-04-03 14:01:14.003549] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=12][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:14.003559] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:14.003567] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:14.003576] WDIAG [SQL] open (ob_result_set.cpp:158) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:14.003585] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:14.003592] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:14.003611] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=18][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:14.003623] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=11][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:14.003651] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=21][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:14.003658] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:14.003666] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:14.003797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.003919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.004036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.004209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.004450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.004562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.004748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.004949] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:14.005248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.005367] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:14.005415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.005528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.005673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.005849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.006113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.006370] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.006390] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.006550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.006400] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] [PALF_DUMP](palf_id=1001, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1001, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:29361635, max_lsn:3574142328, max_scn:{val:1743655277355487643, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:3574142328}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:3532199898}}, last_submit_lsn:{lsn:3574142206}, last_submit_end_lsn:{lsn:3574142328}, last_submit_log_id:29361635, last_submit_log_pid:637, max_flushed_lsn:{lsn:3574142206}, max_flushed_end_lsn:{lsn:3574142328}, max_flushed_log_pid:637, committed_end_lsn:{lsn:3574142328}, last_slide_log_id:29361635, last_slide_scn:{val:1743655277355487643, v:0}, last_slide_lsn:{lsn:3574142206}, last_slide_end_lsn:{lsn:3574142328}, last_slide_log_pid:637, last_slide_log_accum_checksum:3441221005, last_fetch_end_lsn:{lsn:3616085368}, last_fetch_max_log_id:29363683, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660073509082, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7378b5e530}, [StateMgr]={this:0x2b7378b62a30, self:"172.20.0.5:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:640}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:29361636, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2b7378b692f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246113908, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:637, prev:{version:1, config:{config_version:{proposal_id:637, config_seq:421}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246113908, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:6, lock_time:1743590092982597}}, curr:{version:1, config:{config_version:{proposal_id:637, config_seq:422}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246113908, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:6, lock_time:1743590092982597}}, prev_log_proposal_id:637, prev_lsn:{lsn:3565944170}, prev_mode_pid:637}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:637, config_seq:422}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660073706268, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660073706268, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b7378b5fc70}, [ModeMgr]={palf_id:1001, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:637, mode_version:0, access_mode:1, ref_scn:{val:1737616926591978738, v:0}}, accepted_mode_meta:{version:1, proposal_id:637, mode_version:0, access_mode:1, ref_scn:{val:1737616926591978738, v:0}}, last_submit_mode_meta:{version:1, proposal_id:637, mode_version:0, access_mode:1, ref_scn:{val:1737616926591978738, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:3489447936}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:640}, log_config_meta:{version:3, proposal_id:637, prev:{version:1, config:{config_version:{proposal_id:637, config_seq:421}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246113908, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:6, lock_time:1743590092982597}}, curr:{version:1, config:{config_version:{proposal_id:637, config_seq:422}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246113908, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:6, lock_time:1743590092982597}}, prev_log_proposal_id:637, prev_lsn:{lsn:3565944170}, prev_mode_pid:637}, log_snapshot_meta:{version:1, base_lsn:{lsn:3489447936}, prev_log_info:{log_id:28665839, lsn:{lsn:3489443776}, scn:{val:1743497079610912559, v:0}, log_proposal_id:632, accum_checksum:3302889752}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:637, mode_version:0, access_mode:1, ref_scn:{val:1737616926591978738, v:0}}}, log_meta_storage:{log_tail:{lsn:28672}, readable_log_tail:{lsn:28672}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:3623657472}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1001/meta", dir_fd:119, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67076096, block_header_serialize_buf_:0x2b7378b64bb8, flashback_version:0}, log_storage:{log_tail:{lsn:3574142328}, readable_log_tail:{lsn:3574142328}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:3556552704}, min_scn:{val:1743640332268118442, v:0}, curr_block_id:53, palf_id:1001}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1001/log", dir_fd:120, min_block_id:52, max_block_id:54, curr_writable_block_id:53}, logical_block_size_:67104768, curr_block_writable_size_:49515144, block_header_serialize_buf_:0x2b7378b66838, flashback_version:0}, palf_epoch:6, last_purge_throttling_ts:-1, this:0x2b7378b63170}, [Reconfirm]={palf_id:1001, self:"172.20.0.5:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b7378b62bb0}) [2025-04-03 14:01:14.006602] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=197][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1001, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:14.006612] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1001, self="172.20.0.5:2882", ack_info_list=[]) [2025-04-03 14:01:14.006619] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1001, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:14.006625] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.006634] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.007376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.007618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.008089] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.008111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.008576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.008774] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.009103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.009402] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.009413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.010185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.010198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.010311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.010462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.010775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.011124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.011213] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.011242] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.011254] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660074010770], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.011277] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660074010770], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.011310] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660074010770]}) [2025-04-03 14:01:14.011406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.011529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.011718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.012141] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.012167] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.012197] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660074011333], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.012223] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660074011333], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.012250] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660074011333]}) [2025-04-03 14:01:14.012431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.012744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.012943] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.012966] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.012979] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660074012266], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.013003] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660074012266], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.013039] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660074012266]}) [2025-04-03 14:01:14.013220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.013347] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=10][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.013387] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=38][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.013440] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.013426] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=14][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.013456] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=28][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.4:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.013596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.013714] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.013727] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.013740] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.013754] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660074013059], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.013775] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660074013059], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.014242] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.014270] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.014281] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=9][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.014311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.014317] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.014322] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=40][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.3:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.014509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.014516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.014688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.014824] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.014841] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.014868] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:14.014878] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=8][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:14.014888] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=8][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:14.014870] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.014895] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:14.014903] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=7][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:14.014909] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=5][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:14.014901] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660074014181], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.014916] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6][errcode=-4013] query failed(ret=-4013, conn=0x2b73992b8050, start=1743660074001075, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:14.014930] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=13][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:14.014930] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660074014181], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.014937] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=5][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:14.014950] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660074014181]}) [2025-04-03 14:01:14.014989] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:14.014996] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6][errcode=-4013] fail to get ls infos by tenant(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, inner_table_only=false) [2025-04-03 14:01:14.015003] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=6][errcode=-4013] fail to open iterator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.015009] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D5-0-0] [lt=5][errcode=-4013] ls table iterator next failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.015538] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.015578] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=37][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.015599] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=19] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.015647] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074015645, replica_locations:[]}) [2025-04-03 14:01:14.015701] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=49][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.015730] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=27][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074011290}) [2025-04-03 14:01:14.015755] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=23][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.015773] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA699-0-0] [lt=17][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.017036] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.023822] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.023838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.023846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024440] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.024886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.025147] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.025351] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.025560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.025564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.025762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.025986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.026613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.026792] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.026960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.027215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.027374] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.027909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.028202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.028349] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.028497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.028641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.029553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.029857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.030013] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.030201] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.030357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.030383] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.030546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.030945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.031556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.032108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.032697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.033467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.033842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.034344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.034487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.034980] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.035181] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.036354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.037098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.037329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.037744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.037908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.038095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.038235] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.038604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.039678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.040043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.040050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.040250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.040486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.040754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.041688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.042486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.042693] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1548-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.046084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=4] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.046119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.046383] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.046407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.046608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.046631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.046867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.046889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.047366] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.047582] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.047924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.048010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=85] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.048506] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.048656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.048845] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.049475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.050445] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.057763] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.059508] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:8591, tid:17463}]) [2025-04-03 14:01:14.063633] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=12][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.063705] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=67] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.063751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.064233] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.064268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.064281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.064490] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.064548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=55] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.064570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.064928] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.064966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.064980] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.065172] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.065198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.065215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.065470] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.065499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.065510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.065692] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.065729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.065747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.065914] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.066240] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.066289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=46] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.066476] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.066624] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.066651] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.066778] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.066940] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.067123] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.067312] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.067522] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.067709] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.067939] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:14.068178] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4808395022, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:14.070998] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.076920] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.077359] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.077401] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.077430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.077475] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.077649] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.077676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.077694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.077950] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.077992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.078015] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.078210] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.078235] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.078245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.078477] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.078518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.078541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.078595] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.078710] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.078738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.078756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.079101] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.079134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.079148] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.079375] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.079397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.079418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.079602] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.079623] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.079636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.079905] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.079934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.079946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.080187] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.080399] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.080427] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.080459] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.080518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=54] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.080652] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.080782] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.080784] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.080804] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.080935] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.081078] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.081103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.081252] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.081433] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.081442] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.081573] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.081757] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.081952] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.081982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.082101] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.082131] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.082332] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.082422] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.082472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.082543] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.082638] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:14.082632] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.082721] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.082731] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.082747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.082787] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=2][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.082841] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.082842] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.082964] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.082996] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.083069] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.083168] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:14.083476] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4676497979, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:14.083539] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=56] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.089589] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=5) [2025-04-03 14:01:14.106488] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.106507] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.106516] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.106534] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.115835] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.115870] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660074115605]}) [2025-04-03 14:01:14.116145] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660074115882]}) [2025-04-03 14:01:14.116372] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660074116159]}) [2025-04-03 14:01:14.116390] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:14.116399] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:14.116531] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660074116384]}) [2025-04-03 14:01:14.116865] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:14.116883] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=17][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:14.116984] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660074116716]}) [2025-04-03 14:01:14.117099] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.117315] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:14.117329] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:14.117336] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=6][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:14.117454] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=9][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:14.117464] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=10][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:14.117469] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=4][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.4:2882") [2025-04-03 14:01:14.117475] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=5][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.3:2882") [2025-04-03 14:01:14.117478] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=3][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.5:2882") [2025-04-03 14:01:14.117486] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=3][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.117494] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.117512] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=17] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.117524] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=7] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.117536] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=5] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074117534, replica_locations:[]}) [2025-04-03 14:01:14.117556] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=18][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.117563] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=7][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074115858}) [2025-04-03 14:01:14.117570] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=7][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.117575] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69A-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.145923] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4598721898, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.145947] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.146096] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1001}, delta=4796789955, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:14.146111] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.146408] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1002}, delta=4809077211, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:14.146428] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.146693] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1003}, delta=4594663273, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:14.146711] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.150856] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-04-03 14:01:14.150877] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=11] get_disk_usage(ret=0, capacity(MB):=9216, used(MB):=563) [2025-04-03 14:01:14.150979] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=13] cannot_recycle_log_size statistics(cannot_recycle_log_size=322182499, threshold=2899102924, need_update_checkpoint_scn=false) [2025-04-03 14:01:14.157841] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.163694] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4808490536, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:14.168638] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.168654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.168660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.169040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.169148] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.169305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.169931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.170447] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.170504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.170754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.170913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.171066] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.171244] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.171407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.171562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.171885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.172052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.172404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.172550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.172871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.173552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.173681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.174231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.174495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.174821] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.174986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.175178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.175513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.175628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.175849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.175974] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.176123] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.176284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.176622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.176947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.177008] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.177055] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.177066] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.177137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.177257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.177443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.177755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.178087] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.178623] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.194286] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [17813][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=45] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.198743] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1098) [17923][T1001_TabletSta][T1001][Y0-0000000000000000-0-0] [lt=31078] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1001, interval_step=1, last_update_time=1743660074198630) [2025-04-03 14:01:14.206571] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.206648] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=64][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.206667] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.206686] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.209505] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F9-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.215136] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2F9-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.217137] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.218067] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.218159] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660074217293]}) [2025-04-03 14:01:14.219861] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DA-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.220484] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69B-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.220530] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69B-0-0] [lt=43][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.220573] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69B-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074220571, replica_locations:[]}) [2025-04-03 14:01:14.220645] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69B-0-0] [lt=68][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.220672] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69B-0-0] [lt=25][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074218120}) [2025-04-03 14:01:14.220693] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69B-0-0] [lt=19][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.220743] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69B-0-0] [lt=49][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.220759] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28378 [2025-04-03 14:01:14.222403] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DA-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.224316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DA-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.224543] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DA-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.224567] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DA-0-0] [lt=23][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.226139] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.245348] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.245376] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=26][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.245384] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.245398] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.245412] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=6] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074245411, replica_locations:[]}) [2025-04-03 14:01:14.245439] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=25][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.245458] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=19][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074241956}) [2025-04-03 14:01:14.245468] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=9][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.245475] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69C-0-0] [lt=6][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.245936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.246170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.246337] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.246440] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.246548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.246723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.247974] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.248656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.249399] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4598824856, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.252727] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE374-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.257904] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.268609] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28386 [2025-04-03 14:01:14.271160] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.277068] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.277099] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.277111] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.278709] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.280631] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.280681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.280747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.280797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.280905] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.280994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.281104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.281964] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.285267] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.285319] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=44] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28392 [2025-04-03 14:01:14.285422] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28392 [2025-04-03 14:01:14.285451] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.289714] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=39] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=4) [2025-04-03 14:01:14.290943] INFO [CLOG] operator() (ob_log_replay_service.cpp:1589) [18589][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=1019] get_replay_process success(id={id:1}, replayed_log_size=88556798, unreplayed_log_size=0) [2025-04-03 14:01:14.290991] INFO [CLOG] operator() (ob_log_replay_service.cpp:1589) [18589][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=40] get_replay_process success(id={id:1001}, replayed_log_size=84694392, unreplayed_log_size=0) [2025-04-03 14:01:14.291007] INFO [CLOG] operator() (ob_log_replay_service.cpp:1589) [18589][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=11] get_replay_process success(id={id:1002}, replayed_log_size=74011080, unreplayed_log_size=0) [2025-04-03 14:01:14.291046] INFO [CLOG] operator() (ob_log_replay_service.cpp:1589) [18589][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=9] get_replay_process success(id={id:1003}, replayed_log_size=74920229, unreplayed_log_size=0) [2025-04-03 14:01:14.291105] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [18589][T1004_ReplayPro][T1004][Y0-0000000000000000-0-0] [lt=10] dump tenant replay process(tenant_id=1004, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=307, last_replayed_log_size(MB)=307, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-04-03 14:01:14.300960] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.301008] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28394 [2025-04-03 14:01:14.301119] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28394 [2025-04-03 14:01:14.301137] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.302153] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-04-03 14:01:14.302194] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=33] get_disk_usage(ret=0, capacity(MB):=1024, used(MB):=255) [2025-04-03 14:01:14.302231] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=14] cannot_recycle_log_size statistics(cannot_recycle_log_size=200526433, threshold=322122547, need_update_checkpoint_scn=false) [2025-04-03 14:01:14.306699] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.306741] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=36][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.306758] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.306772] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.317234] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.322215] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=91] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:14.325407] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660074324640]}) [2025-04-03 14:01:14.326107] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660074325448]}) [2025-04-03 14:01:14.326635] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660074326132]}) [2025-04-03 14:01:14.327177] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660074326658]}) [2025-04-03 14:01:14.328227] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660074327512]}) [2025-04-03 14:01:14.328362] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.328399] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=35][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.328418] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=17] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.328449] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=19] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.328475] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074328473, replica_locations:[]}) [2025-04-03 14:01:14.328539] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=60][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.328559] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=19][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074325375}) [2025-04-03 14:01:14.328579] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=18][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.328592] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69D-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.346275] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4598922560, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.346324] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.346487] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1001}, delta=4796990617, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:14.346515] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.346900] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1002}, delta=4809277873, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:14.346933] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.347335] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1003}, delta=4594863956, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:14.347364] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.357998] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.363663] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4808690273, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:14.363716] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=46] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.371241] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.373668] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.373690] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=21][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.373705] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.373718] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=7] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.373739] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074373735, replica_locations:[]}) [2025-04-03 14:01:14.373760] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=19][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.373769] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=8][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074370267}) [2025-04-03 14:01:14.373777] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=7][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.373781] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69E-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.377139] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.377174] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.377181] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.378790] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.380769] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=88] get wrs ts(ls_id={id:1}, delta=4676794404, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:14.380853] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=76] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.406791] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.406856] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=58][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.406878] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.406896] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.412143] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=26] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.412379] INFO [MDS] flush (mds_table_mgr.cpp:172) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=1] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b73b7ede510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1743655263788752080, v:0}, clog_base_lsn:{lsn:3690762240}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743648258823515129, v:0}, all_id_meta:{id_meta:[{limited_id:1740569710308384976, latest_log_ts:{val:1740569693675557106, v:0}}, {limited_id:32000001, latest_log_ts:{val:1740551265169597432, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:1292, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:14.417314] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.419352] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.419386] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=28] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28396 [2025-04-03 14:01:14.419465] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28396 [2025-04-03 14:01:14.419480] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.424226] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.424284] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=48] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28398 [2025-04-03 14:01:14.424383] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28398 [2025-04-03 14:01:14.424400] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.430491] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660074429914]}) [2025-04-03 14:01:14.431267] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660074430534]}) [2025-04-03 14:01:14.432711] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660074431302]}) [2025-04-03 14:01:14.432842] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.432881] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=38][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.432920] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=37] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.432950] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=18] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.433001] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=37] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074432998, replica_locations:[]}) [2025-04-03 14:01:14.433081] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=75][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.433101] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=18][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074430459}) [2025-04-03 14:01:14.433120] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=17][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.433134] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA69F-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.434521] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660074432743]}) [2025-04-03 14:01:14.437044] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660074434938]}) [2025-04-03 14:01:14.438202] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17808][T1_RFLWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.449686] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.449831] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28400 [2025-04-03 14:01:14.450133] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4599025972, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.458170] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=65][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.462655] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE7B-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.467720] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.471305] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.472830] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.475261] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.477245] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.477304] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.477324] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.478870] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.489850] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=3) [2025-04-03 14:01:14.500615] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.506980] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.507108] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=36][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.508334] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.508384] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.508412] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.508430] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.516246] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.516289] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28414 [2025-04-03 14:01:14.516395] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=32] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28414 [2025-04-03 14:01:14.516436] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.517392] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.526551] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:14.526580] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28416 [2025-04-03 14:01:14.526653] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28416 [2025-04-03 14:01:14.526671] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.539183] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660074537851]}) [2025-04-03 14:01:14.540848] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660074539225]}) [2025-04-03 14:01:14.541319] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660074540884]}) [2025-04-03 14:01:14.543170] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.543213] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=41][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.543228] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.543251] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.543276] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074543273, replica_locations:[]}) [2025-04-03 14:01:14.543338] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=57][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.543364] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=24][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074539141}) [2025-04-03 14:01:14.543386] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=20][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.543400] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A0-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.543733] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660074541348]}) [2025-04-03 14:01:14.545639] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660074544119]}) [2025-04-03 14:01:14.546155] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1004, server_version={version:{val:1743659606093488000, v:0}, total_part_count:4, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=468452662, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.20.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=56576, succ_cluster_heartbeat_count=56576, cluster_heartbeat_interval=1000000, local_cluster_version={val:1743657204488487203, v:0}, local_cluster_delta=2870057663, force_self_check=false, weak_read_refresh_interval=100000) [2025-04-03 14:01:14.546417] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=75] get wrs ts(ls_id={id:1}, delta=4599122191, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.547007] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.547072] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=58] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.558255] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=59][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.571377] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.572141] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.572161] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28418 [2025-04-03 14:01:14.572191] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28418 [2025-04-03 14:01:14.572198] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.577335] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.577390] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.577410] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.578944] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.584986] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.585047] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=53] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28420 [2025-04-03 14:01:14.585131] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28420 [2025-04-03 14:01:14.585174] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.593118] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.593143] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28422 [2025-04-03 14:01:14.593212] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28422 [2025-04-03 14:01:14.593236] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.604207] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.607161] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.607222] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=54][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.607270] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.608577] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1003, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:74920229}, prev_lsn={lsn:74920107}, fetch_start_log_id=615432, last_slide_log_id=615431, fetch_log_size=41943040, accepted_mode_pid=14, last_fetch_req_time=1743660074608575, last_fetch_end_lsn={lsn:116863269}, last_fetch_max_log_id=617479, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1003, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:615432, max_lsn:74920351, max_scn:{val:1743655479584546174, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:74920351}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:74671177}}, last_submit_lsn:{lsn:74920229}, last_submit_end_lsn:{lsn:74920351}, last_submit_log_id:615432, last_submit_log_pid:14, max_flushed_lsn:{lsn:74920229}, max_flushed_end_lsn:{lsn:74920351}, max_flushed_log_pid:14, committed_end_lsn:{lsn:74920229}, last_slide_log_id:615431, last_slide_scn:{val:1743655479483236415, v:0}, last_slide_lsn:{lsn:74920107}, last_slide_end_lsn:{lsn:74920229}, last_slide_log_pid:14, last_slide_log_accum_checksum:2102179633, last_fetch_end_lsn:{lsn:116863269}, last_fetch_max_log_id:617479, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660074608575, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7372ff0530}) [2025-04-03 14:01:14.608704] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=121][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.617485] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=55][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.620645] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.626908] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17558][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:14.629046] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.638967] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:14.641036] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.641128] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28426 [2025-04-03 14:01:14.641878] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [17699][T1_PalfGC][T1][Y0-0000000000000000-0-0] [lt=25] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.648479] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A1-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.648509] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A1-0-0] [lt=29][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.648537] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A1-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074648535, replica_locations:[]}) [2025-04-03 14:01:14.648577] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A1-0-0] [lt=26][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.648604] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A1-0-0] [lt=26][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074647227}) [2025-04-03 14:01:14.648615] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A1-0-0] [lt=10][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.648621] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A1-0-0] [lt=6][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.648826] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660074647946]}) [2025-04-03 14:01:14.649645] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660074648850]}) [2025-04-03 14:01:14.649846] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.649872] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28428 [2025-04-03 14:01:14.649921] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28428 [2025-04-03 14:01:14.650549] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4599226840, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.650960] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1002}, delta=4809582153, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:14.652972] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28430 [2025-04-03 14:01:14.658730] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.662157] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [18377][T1003_DagSchedu][T1003][Y0-0000000000000000-0-0] [lt=1300] dump_dag_status(dag_cnt=0, map_size=0) [2025-04-03 14:01:14.662175] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1990) [18377][T1003_DagSchedu][T1003][Y0-0000000000000000-0-0] [lt=17] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.662749] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.662794] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:23016147553}, prev_lsn={lsn:23016147012}, fetch_start_log_id=40888245, last_slide_log_id=40888244, fetch_log_size=41943040, accepted_mode_pid=620, last_fetch_req_time=1743660074662792, last_fetch_end_lsn={lsn:23058090593}, last_fetch_max_log_id=40890292, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:40888393, max_lsn:23016225752, max_scn:{val:1743655278027912055, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:23016148094}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:22974282712}}, last_submit_lsn:{lsn:23016147553}, last_submit_end_lsn:{lsn:23016148094}, last_submit_log_id:40888245, last_submit_log_pid:620, max_flushed_lsn:{lsn:23016147553}, max_flushed_end_lsn:{lsn:23016148094}, max_flushed_log_pid:620, committed_end_lsn:{lsn:23016147553}, last_slide_log_id:40888244, last_slide_scn:{val:1743655265673135133, v:0}, last_slide_lsn:{lsn:23016147012}, last_slide_end_lsn:{lsn:23016147553}, last_slide_log_pid:620, last_slide_log_accum_checksum:2388580578, last_fetch_end_lsn:{lsn:23058090593}, last_fetch_max_log_id:40890292, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660074662792, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b734c5f2530}) [2025-04-03 14:01:14.662903] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=100][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.664551] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.667847] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.668125] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=2] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.671467] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.677425] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.677498] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.677551] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.679698] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=34][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.681571] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.681604] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.682232] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.682340] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4677096261, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:14.684400] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28436 [2025-04-03 14:01:14.687498] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.689975] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=30] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=2) [2025-04-03 14:01:14.701638] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.701699] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=53] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28440 [2025-04-03 14:01:14.701787] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28440 [2025-04-03 14:01:14.701833] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=41] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.705069] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18594][T1004_RFLWorker][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.705387] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=32] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:14.707258] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.707319] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=54][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.707348] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.707366] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.717574] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.722370] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.722414] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28442 [2025-04-03 14:01:14.722499] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28442 [2025-04-03 14:01:14.722522] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.724369] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=78] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:14.735487] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=34][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:14.735511] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=23][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660075703884, now=[mts=1743660073525315], now0=[mts=1743660073525315], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:14.735534] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a4335d80, tx_id:{txid:76175010}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660073520466, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:14.735591] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=52][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73961321a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a4335d80}) [2025-04-03 14:01:14.735616] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=24][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a4335d80, tx_id:{txid:76175010}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660073520466, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:14.735648] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=29] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f8054008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660073442684, tx_desc={this:0x2b73a4335d80, tx_id:{txid:76175010}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660073520466, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660045773894, use_das=false, nested_level=0, session={this:0x2b73961321a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a4335d80}, plan=0x2b7338b48050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:14.735722] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=72][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:14.735740] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:14.735749] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:14.735758] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=7][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:14.735767] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=7][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:14.735785] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=16][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=22, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:14.735800] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12] will sleep(sleep_us=22000, remain_us=1038085, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=22, v.err_=-4283, timeout_timestamp=1743660075773884) [2025-04-03 14:01:14.746605] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4599322435, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.746786] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1001}, delta=4797390492, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:14.746950] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1002}, delta=4809677748, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:14.747123] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=6][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.747148] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=23][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.747153] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1003}, delta=4595263832, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:14.747644] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=40] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1003}, {id:1}, {id:1002}]) [2025-04-03 14:01:14.749589] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.749620] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=29][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.749630] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.749660] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074749658, replica_locations:[]}) [2025-04-03 14:01:14.749702] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=39][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.749714] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=11][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074745672}) [2025-04-03 14:01:14.749725] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=10][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.749731] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A2-0-0] [lt=5][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.753910] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660074753387]}) [2025-04-03 14:01:14.755112] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.755142] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=28][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.755151] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.755167] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.755181] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074755180, replica_locations:[]}) [2025-04-03 14:01:14.755222] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=39][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.755233] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=10][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074752220}) [2025-04-03 14:01:14.755242] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=8][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.755248] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A3-0-0] [lt=5][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.755315] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660074754349]}) [2025-04-03 14:01:14.757905] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=29][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:22, local_retry_times:22, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:14.757953] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=31][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:14.757960] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=6][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:14.757964] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=4][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:14.758015] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=5][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.758452] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.758947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.759892] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.759916] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:14.760608] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:14.761112] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:14802, tid:17514}]) [2025-04-03 14:01:14.764534] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=63] get wrs ts(ls_id={id:1}, delta=4809090679, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:14.764566] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.768946] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.768985] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28444 [2025-04-03 14:01:14.769100] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28444 [2025-04-03 14:01:14.769117] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.771577] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.777512] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.777571] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.777840] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.777865] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.778114] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.778135] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.778467] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.778495] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.778684] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.778698] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.779067] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.779090] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.779254] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.779273] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.779526] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.779546] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.779700] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.781165] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.781198] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.781277] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.781301] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.781380] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.781400] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.781475] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.781494] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.781657] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.781677] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.781842] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.781868] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.781965] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.781985] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782073] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.782090] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782302] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.782322] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782387] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.782397] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.782410] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782411] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782554] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.782564] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782663] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.782683] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782681] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.782696] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782796] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.782833] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782884] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.782903] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.782989] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.783014] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.783009] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.783052] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.783150] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.783161] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.783235] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.783260] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:14.783267] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.783273] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.783352] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=4677197017, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:14.783391] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.783490] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.783503] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.783802] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.783840] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.784270] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.784294] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.784640] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:14.784662] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:14.784748] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1001, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:3727091690}, prev_lsn={lsn:3727091577}, fetch_start_log_id=30619388, last_slide_log_id=30619387, fetch_log_size=41943040, accepted_mode_pid=1292, last_fetch_req_time=1743660074784747, last_fetch_end_lsn={lsn:3769034730}, last_fetch_max_log_id=30621435, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1001, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:30619539, max_lsn:3727110187, max_scn:{val:1743655279791409480, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:3727091690}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:3690762240}}, last_submit_lsn:{lsn:3727091577}, last_submit_end_lsn:{lsn:3727091690}, last_submit_log_id:30619387, last_submit_log_pid:1292, max_flushed_lsn:{lsn:3727091577}, max_flushed_end_lsn:{lsn:3727091690}, max_flushed_log_pid:1292, committed_end_lsn:{lsn:3727091690}, last_slide_log_id:30619387, last_slide_scn:{val:1743655263788752080, v:0}, last_slide_lsn:{lsn:3727091577}, last_slide_end_lsn:{lsn:3727091690}, last_slide_log_pid:1292, last_slide_log_accum_checksum:3616297708, last_fetch_end_lsn:{lsn:3769034730}, last_fetch_max_log_id:30621435, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660074784747, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7333f70530}) [2025-04-03 14:01:14.784879] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=121] [PALF_DUMP](palf_id=1001, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1001, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:30619539, max_lsn:3727110187, max_scn:{val:1743655279791409480, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:3727091690}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:3690762240}}, last_submit_lsn:{lsn:3727091577}, last_submit_end_lsn:{lsn:3727091690}, last_submit_log_id:30619387, last_submit_log_pid:1292, max_flushed_lsn:{lsn:3727091577}, max_flushed_end_lsn:{lsn:3727091690}, max_flushed_log_pid:1292, committed_end_lsn:{lsn:3727091690}, last_slide_log_id:30619387, last_slide_scn:{val:1743655263788752080, v:0}, last_slide_lsn:{lsn:3727091577}, last_slide_end_lsn:{lsn:3727091690}, last_slide_log_pid:1292, last_slide_log_accum_checksum:3616297708, last_fetch_end_lsn:{lsn:3769034730}, last_fetch_max_log_id:30621435, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660074784747, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7333f70530}, [StateMgr]={this:0x2b7333f74a30, self:"172.20.0.5:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1292}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:30619388, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2b7333f7b2f0, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648245973413, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:1292, prev:{version:1, config:{config_version:{proposal_id:1292, config_seq:451}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648245973413, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1292, config_seq:452}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648245973413, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:1292, prev_lsn:{lsn:3718852503}, prev_mode_pid:1292}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:1292, config_seq:452}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660070967313, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660070967313, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b7333f71c70}, [ModeMgr]={palf_id:1001, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:1292, mode_version:0, access_mode:1, ref_scn:{val:1737450300811649224, v:0}}, accepted_mode_meta:{version:1, proposal_id:1292, mode_version:0, access_mode:1, ref_scn:{val:1737450300811649224, v:0}}, last_submit_mode_meta:{version:1, proposal_id:1292, mode_version:0, access_mode:1, ref_scn:{val:1737450300811649224, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:3690762240}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:1292}, log_config_meta:{version:3, proposal_id:1292, prev:{version:1, config:{config_version:{proposal_id:1292, config_seq:451}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648245973413, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:1292, config_seq:452}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648245973413, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:1292, prev_lsn:{lsn:3718852503}, prev_mode_pid:1292}, log_snapshot_meta:{version:1, base_lsn:{lsn:3690762240}, prev_log_info:{log_id:30320912, lsn:{lsn:3690758089}, scn:{val:1743624467520100647, v:0}, log_proposal_id:1292, accum_checksum:1072638919}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:1292, mode_version:0, access_mode:1, ref_scn:{val:1737450300811649224, v:0}}}, log_meta_storage:{log_tail:{lsn:24576}, readable_log_tail:{lsn:24576}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:3757867008}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1002/1001/meta", dir_fd:105, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67080192, block_header_serialize_buf_:0x2b7333f76bb8, flashback_version:0}, log_storage:{log_tail:{lsn:3727091690}, readable_log_tail:{lsn:3727091690}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:3690762240}, min_scn:{val:1743624467520100648, v:0}, curr_block_id:55, palf_id:1001}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1002/1001/log", dir_fd:106, min_block_id:55, max_block_id:56, curr_writable_block_id:55}, logical_block_size_:67104768, curr_block_writable_size_:30775318, block_header_serialize_buf_:0x2b7333f78838, flashback_version:0}, palf_epoch:6, last_purge_throttling_ts:-1, this:0x2b7333f75170}, [Reconfirm]={palf_id:1001, self:"172.20.0.5:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b7333f74bb0}) [2025-04-03 14:01:14.785306] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=419][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1001, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:14.785333] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25] [PALF_DUMP](palf_id=1001, self="172.20.0.5:2882", ack_info_list=[]) [2025-04-03 14:01:14.785355] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1001, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:14.785370] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4308) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] current log_task status(palf_id=1001, self="172.20.0.5:2882", log_id=30619388, log_task={header:{begin_lsn:{lsn:18446744073709551615}, end_lsn:{lsn:18446744073709551615}, log_id:-1, min_scn:{val:18446744073709551615, v:3}, max_scn:{val:18446744073709551615, v:3}, data_len:0, proposal_id:9223372036854775807, prev_lsn:{lsn:18446744073709551615}, prev_proposal_id:9223372036854775807, committed_end_lsn:{lsn:18446744073709551615}, data_checksum:-1, accum_checksum:-1, is_padding_log:false, is_raw_write:false}, state_map:{val:0}, ref_cnt:0, gen_ts:-1, freeze_ts:-1, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:0, submit_to_flush cost time:0}, ack_info_list=[]) [2025-04-03 14:01:14.785413] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.793938] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=47] PNIO [ratelimit] time: 1743660074793937, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006061, add_bytes: 0 [2025-04-03 14:01:14.803681] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=19] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.807337] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.807383] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.807402] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.807416] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.810255] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.814858] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=21][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:14.814899] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=40][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660073619117], now0=[mts=1743660073619117], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:14.814965] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=62][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719467}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660073617776, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:14.815072] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=100][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:14.815110] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=36][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719467}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660073617776, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:14.815161] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=47] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.815256] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=93][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:14.815279] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:14.815303] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=22][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:14.815318] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=13][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:14.815331] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:14.815350] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=17][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=16, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:14.817622] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.819780] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.819799] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28446 [2025-04-03 14:01:14.819838] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28446 [2025-04-03 14:01:14.819855] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.831468] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:16, local_retry_times:16, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:14.831542] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=53][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:14.831560] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=16][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:14.831586] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=24][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:14.831613] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=13][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:14.834422] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1962) [18519][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=6884] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.834567] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.834648] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:14.841509] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2449) [18519][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=0] Dumping All Cache Objs(alloc_obj_list.count()=70, alloc_obj_list=[{obj_id:1, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175445}, {obj_id:2, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176543}, {obj_id:3, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:436996}, {obj_id:4, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:309921}, {obj_id:5, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:251066}, {obj_id:6, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:305539}, {obj_id:7, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:246674}, {obj_id:8, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175828}, {obj_id:11, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:179377}, {obj_id:12, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180892}, {obj_id:13, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176578}, {obj_id:15, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183126}, {obj_id:17, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224434}, {obj_id:18, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:227956}, {obj_id:19, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195529}, {obj_id:22, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173048}, {obj_id:23, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181961}, {obj_id:24, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191770}, {obj_id:25, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187525}, {obj_id:26, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196150}, {obj_id:27, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173385}, {obj_id:28, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:303583}, {obj_id:29, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223881}, {obj_id:30, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:230702}, {obj_id:31, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:216730}, {obj_id:32, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:223528}, {obj_id:33, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198295}, {obj_id:34, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198293}, {obj_id:35, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192308}, {obj_id:36, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:213222}, {obj_id:37, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:197338}, {obj_id:38, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:198558}, {obj_id:39, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188271}, {obj_id:40, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186147}, {obj_id:41, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:184862}, {obj_id:42, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:199377}, {obj_id:43, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:192389}, {obj_id:44, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187646}, {obj_id:45, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:46, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188001}, {obj_id:47, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185069}, {obj_id:48, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187994}, {obj_id:49, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185011}, {obj_id:50, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191261}, {obj_id:51, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186550}, {obj_id:52, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182609}, {obj_id:53, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209465}, {obj_id:54, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182625}, {obj_id:55, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187423}, {obj_id:56, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183025}, {obj_id:57, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191136}, {obj_id:58, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182721}, {obj_id:59, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183848}, {obj_id:60, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:4992223}, {obj_id:61, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253856}, {obj_id:62, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:3708891}, {obj_id:63, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:245364}, {obj_id:64, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188567}, {obj_id:65, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174057}, {obj_id:66, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178338}, {obj_id:67, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244258}, {obj_id:68, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220628}, {obj_id:69, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274877}, {obj_id:70, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214174}, {obj_id:71, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205532}, {obj_id:72, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196291}, {obj_id:73, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190887}, {obj_id:74, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176654}, {obj_id:75, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:195529}, {obj_id:76, tenant_id:1004, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212788}]) [2025-04-03 14:01:14.841866] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2702) [18519][T1004_PlanCache][T1004][Y0-0000000000000000-0-0] [lt=349] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:14.848139] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [17564][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=40] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:14.849160] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:14.849181] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28448 [2025-04-03 14:01:14.849223] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28448 [2025-04-03 14:01:14.849232] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.850940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.851250] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:14.851257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.851536] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:14.851692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.852009] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.852216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.852366] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.852417] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.852559] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4599428545, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.852599] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.852617] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=9] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:14.852775] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=38] get wrs ts(ls_id={id:1001}, delta=4797496602, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:14.852804] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.852931] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=51] get wrs ts(ls_id={id:1002}, delta=4809783858, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:14.852957] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.853092] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=37] get wrs ts(ls_id={id:1003}, delta=4595368853, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:14.853120] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.854847] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.854894] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28450 [2025-04-03 14:01:14.854975] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28450 [2025-04-03 14:01:14.855006] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.856535] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=11182][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.856589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=50] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.856717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.857166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.857340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.857609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.857766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.857930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.858036] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660074856100]}) [2025-04-03 14:01:14.858474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.858530] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=39][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.858773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.859185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.859342] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660074858083]}) [2025-04-03 14:01:14.859462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.859477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.859864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.859883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.860103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.860118] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.860388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.860406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.860797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.860847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=48] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.861075] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.861093] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.861395] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.861412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.861627] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660074859375]}) [2025-04-03 14:01:14.862012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.862053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.862359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.862394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.863131] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660074861657]}) [2025-04-03 14:01:14.863618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.863641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.863770] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:14.863983] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:108) [17630][OmtNodeBalancer][T0][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] refresh tenant units(sys_unit_cnt=0, units=[], ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.864672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.864696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.864800] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.864831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.864883] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660074863493]}) [2025-04-03 14:01:14.864924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.864934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.865128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.865144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.865348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.865460] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.865632] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.865930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.866109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.866452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.866851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.866870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.867034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.867192] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4809193584, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:14.867515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.867729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.868100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.868330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.868675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.868828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.868915] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A4-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.868937] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A4-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.868966] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A4-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074868964, replica_locations:[]}) [2025-04-03 14:01:14.868996] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A4-0-0] [lt=28][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.869011] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A4-0-0] [lt=14][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074857995}) [2025-04-03 14:01:14.869038] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A4-0-0] [lt=25][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.869046] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A4-0-0] [lt=7][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.869150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.869514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.869665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17630][OmtNodeBalancer][T1][YB42AC140005-000631D6BA8CBE53-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.869975] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:301) [17630][OmtNodeBalancer][T0][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0] cal_all_part_disk_default_percentage succ(data_dir="/data/oceanbase_data/sstable", clog_dir="/data/oceanbase_data/clog", shared_mode=true, data_disk_total_size=193179156480, data_disk_default_percentage=60, clog_disk_total_size=193179156480, clog_disk_default_percentage=30) [2025-04-03 14:01:14.870001] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:132) [17630][OmtNodeBalancer][T0][YB42AC140005-000631D6BA8CBE53-0-0] [lt=0] refresh tenant config(tenants=[], ret=-4013) [2025-04-03 14:01:14.871684] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.873234] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.873287] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=46] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28454 [2025-04-03 14:01:14.873359] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28454 [2025-04-03 14:01:14.873383] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.877601] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.877656] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.877676] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.879768] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.881243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.881278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.881421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.881446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.881562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.881611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.881732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.881755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882927] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.882954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.883006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.883036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.883092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.883177] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4677297727, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:14.884372] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.884401] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28456 [2025-04-03 14:01:14.884454] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28456 [2025-04-03 14:01:14.884469] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.890122] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=62] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=1) [2025-04-03 14:01:14.893891] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.893925] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28458 [2025-04-03 14:01:14.893973] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28458 [2025-04-03 14:01:14.893985] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.901755] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:14.901776] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28460 [2025-04-03 14:01:14.901838] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28460 [2025-04-03 14:01:14.901865] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:14.907404] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.907434] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.907446] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.907458] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] re_register_parent reason(palf_id=1003, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660070802919, last_first_register_time_us=1743648247119503, register_time_us=1743660070802919, curr_time_us=1743660074907456) [2025-04-03 14:01:14.907529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907646] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907826] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.907879] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908536] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908567] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908674] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.908852] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.917687] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:14.946869] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=4599522758, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:14.946929] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=51] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.947157] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=55] get wrs ts(ls_id={id:1001}, delta=4797590815, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:14.947194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.947479] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1002}, delta=4809879168, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:14.947521] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=36] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.947635] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1003}, delta=4595464163, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:14.947666] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.958623] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.961427] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=42][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4235, tid:17828}]) [2025-04-03 14:01:14.965129] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4809291706, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:14.965182] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.967157] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=50][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.967194] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.967209] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660074965717], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.967265] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660074965717], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.967322] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660074965717]}) [2025-04-03 14:01:14.968344] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=14][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.968377] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=31][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.968397] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=15][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.968426] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=28][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.4:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.968599] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.968625] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.968638] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660074967359], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.968639] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=23][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:14.968655] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660074967359], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.968669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.968685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.968708] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660074967359]}) [2025-04-03 14:01:14.968913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.968930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.969159] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}]) [2025-04-03 14:01:14.969368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.969415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=45] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.969716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.969733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.969892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.969908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.970068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.970084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.970233] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:14.970245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.970264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.970428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.970453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.970511] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.970538] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.970550] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660074968727], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.970568] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660074968727], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.970610] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=34] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660074968727]}) [2025-04-03 14:01:14.970642] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:14.970865] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.970883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971582] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971730] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=50][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.971864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.971896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972226] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.972253] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.972266] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660074970630], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.972282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972282] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660074970630], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.972294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972308] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660074970630]}) [2025-04-03 14:01:14.972451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972686] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:14.972873] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.972896] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.972913] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=13][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.972937] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=23][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.3:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.973089] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:14.974340] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.974375] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.974388] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660074972688], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.974406] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660074972688], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:14.974441] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660074972688]}) [2025-04-03 14:01:14.974680] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.974714] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=32][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.974736] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=18][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.974760] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=23][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.5:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.975324] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.975386] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=60][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.975402] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.975452] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=38] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.975482] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=17] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074975480, replica_locations:[]}) [2025-04-03 14:01:14.975529] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=43][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.975550] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=19][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074967282}) [2025-04-03 14:01:14.975570] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=18][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.975582] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A5-0-0] [lt=11][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.977697] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.977777] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=73][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.977799] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.979823] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:14.981103] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:14.981283] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4677395734, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:14.981332] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=40] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:14.981732] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:14.989311] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=12][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.989343] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=30][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.989361] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=14][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.989388] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=26][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.4:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.990435] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.990460] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=24][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.990475] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=13][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.990517] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=41][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.3:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.991308] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:14.991329] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=20][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:14.991343] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=11][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.991363] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=19][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.5:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:14.991851] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.991882] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=29][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:14.991900] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=16] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:14.991927] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:14.991950] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660074991949, replica_locations:[]}) [2025-04-03 14:01:14.992002] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=49][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:14.992044] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=40][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660074988288}) [2025-04-03 14:01:14.992081] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=35][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:14.992094] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A6-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:14.992175] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:14.992843] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.003331] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.004526] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.007486] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.007538] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.007560] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.007578] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.014954] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.015306] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.015326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015572] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.015661] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015667] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015816] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015824] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.015898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016004] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016009] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.016884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017005] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017174] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.017786] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.018474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.018484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.018625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.018631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.018746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.018758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.018880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.018886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.019930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020536] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.020918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021038] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021045] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021121] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021145] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.021158] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=13][errcode=-4013] oops, alloc failed, tenant_id=1001, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=536870900, tenant_hold=397373440, tenant_limit=536870912 [2025-04-03 14:01:15.021166] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.021172] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6][errcode=-4013] rpc memory pool alloc memory failed(sz=1305, alloc_sz=16256) [2025-04-03 14:01:15.021178] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1121) [2025-04-03 14:01:15.021184] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:15.021195] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=29999788, timeout_timestamp=1743660105018174) [2025-04-03 14:01:15.021205] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=9][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:15.021212] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:15.021218] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:15.021224] WDIAG [SQL] open (ob_result_set.cpp:158) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:15.021232] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:15.021241] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=8][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:15.021250] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=9][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:15.021259] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:15.021285] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=15][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:15.021289] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:15.021294] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:15.021408] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.021869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.022995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023579] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023767] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.023981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024189] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024774] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.024976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.025851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026644] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.026698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.026995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027496] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.027569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027635] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.027908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028001] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028007] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028439] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028561] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028663] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028861] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.028979] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029276] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029990] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.029999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030380] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030579] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.030881] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:15.030888] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:15.030897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:15.030908] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=9][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.030914] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:15.030921] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:15.030927] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] query failed(ret=-4013, conn=0x2b7354910050, start=1743660075018173, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.030935] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=7][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:15.030942] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=4][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.030997] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:15.031003] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:104) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] fail to get ls infos by tenant(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, inner_table_only=false) [2025-04-03 14:01:15.031009] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:71) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=6][errcode=-4013] fail to open iterator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.031015] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:334) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] ls table iterator next failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.031033] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:214) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=13][errcode=-4013] build replica map from ls table failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", mode=0) [2025-04-03 14:01:15.031045] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:188) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=9][errcode=-4013] check ls table failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", mode=0) [2025-04-03 14:01:15.031051] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:44) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D6-0-0] [lt=5][errcode=-4013] fail to check ls meta table(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.037898] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.038367] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.045142] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.045158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.045165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.045444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.045452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.045598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.045615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.045915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.045923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046046] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046235] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046241] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=48] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.046998] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047004] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047667] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047689] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.047978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.048736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049059] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049429] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.049591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.049854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050087] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.050837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=211] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051374] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051761] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.051882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052089] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052257] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.052274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052283] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4599628287, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:15.052359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052317] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.052450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052460] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=57] get wrs ts(ls_id={id:1001}, delta=4797696344, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:15.052510] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=45] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.052596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052626] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.052627] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1002}, delta=4809983600, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:15.052647] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.052733] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1003}, delta=4595568595, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:15.052749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052760] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.052887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.052898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053048] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053148] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.053864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054032] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054361] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.054405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054402] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054635] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=47] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.054955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055244] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055253] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=3077][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.055373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.055746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.056109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.056239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.056611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.056717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.056918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.057086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.057192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.057378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.057500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.058005] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.058012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.058206] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.058423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.058582] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.058712] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.058841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.059141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.059492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.059652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.059986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.060176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.060528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.061686] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:7769, tid:17513}]) [2025-04-03 14:01:15.062134] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.062168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.062184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.062858] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.062899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.062915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.063414] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.063434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.063453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.063941] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.063963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.063975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.064644] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.064675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.064687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.065250] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.065273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.065294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.065832] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=50][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.065872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.065884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.066162] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.066180] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.066191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.066418] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.066434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.066442] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.066668] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.066697] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.066723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.066724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=52] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.066732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.066740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.067112] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.067156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.067170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.067356] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.067375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.067410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.067624] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.067640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.067649] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.067665] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.067795] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.067830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.068033] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.068090] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.068137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=42] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.068262] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.068295] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.068419] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.068435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.068470] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.068569] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.068629] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.068637] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.068703] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.068731] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.068765] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.068869] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.068939] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.069044] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.069084] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.069082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.069196] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.069264] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:15.069283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.069345] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D1549-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.069419] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.069630] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4809396148, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.070008] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.070074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=62] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.070099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.070245] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.070458] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.070489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.070510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.070849] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.070900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.071154] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.071214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=56] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.071433] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.071478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=41] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.071692] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.071805] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.071759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=63] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.072143] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.072219] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=71] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.072585] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.072984] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.073039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.073489] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.073556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=62] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.073578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.073734] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.073770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.073962] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.074180] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.074214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.074384] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.074444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=57] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.074590] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.074930] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.074992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=58] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.075236] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.075737] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660075075147]}) [2025-04-03 14:01:15.075764] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.075854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=85] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.075878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.076310] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660075075763]}) [2025-04-03 14:01:15.076393] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.076463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=65] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.076543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=77] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.076774] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660075076329]}) [2025-04-03 14:01:15.077122] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.077162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.077184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.077267] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.077325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=55] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.077510] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.077542] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.077618] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.077727] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.077843] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.077913] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.077932] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.078114] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.078149] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075077682]}) [2025-04-03 14:01:15.078168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=50] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.078317] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.078336] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.078481] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.078605] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.078737] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.078854] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.078963] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.078989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.079039] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.079142] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.079223] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.079643] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.079673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.079713] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.079875] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.079957] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.079997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.080086] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.080099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.080137] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:15.080157] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=16][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:15.080245] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.080699] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.080722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.080759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.080925] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.080947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.081005] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081113] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.081164] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081232] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081271] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.081343] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081359] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081425] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081426] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081477] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081529] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081607] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.081662] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081701] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081714] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081777] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081841] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081857] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081899] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.081960] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.081996] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082048] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082100] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:15.082120] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082115] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:15.082189] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.082274] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082275] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082341] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082434] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082477] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4677496876, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:15.082685] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082763] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082954] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.082974] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.083104] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.083187] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.083290] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.083422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=130] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.083553] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.083574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.083700] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.083787] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.083872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=83] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.083946] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.084043] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.084157] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.084169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.084373] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.084471] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.084487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EB-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.085567] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.085620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=50] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.085633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.085853] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=0][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-04-03 14:01:15.085873] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=17][errcode=0] fail to send rpc(tmp_ret=-4013, tmp_ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, addr="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:15.085889] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.4:2882", cluster_id:1}) [2025-04-03 14:01:15.085950] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.085980] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.085990] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.086095] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.086128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.086323] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.086642] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.086683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.086759] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.086904] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.086943] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.087145] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.3:2882", cluster_id:1}) [2025-04-03 14:01:15.087170] WDIAG [RPC] wait (ob_async_rpc_proxy.h:422) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=24][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.20.0.5:2882", cluster_id:1}) [2025-04-03 14:01:15.087181] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=9][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.4:2882") [2025-04-03 14:01:15.087192] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.3:2882") [2025-04-03 14:01:15.087199] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:315) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=6][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.20.0.5:2882") [2025-04-03 14:01:15.087215] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=6][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.087232] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.087240] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.087294] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=0] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075087291, replica_locations:[]}) [2025-04-03 14:01:15.087330] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=33][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.087344] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=13][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075075720}) [2025-04-03 14:01:15.087357] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=12][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.087366] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A7-0-0] [lt=8][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.087518] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:15.087549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.088166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.088321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.088487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.088653] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.088836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.089144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.089300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.089465] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.089642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.089953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17414][ServerGTimer][T1001][YB42AC140005-000631D6BBFCD1EC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.090243] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.090440] INFO [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:243) [17414][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0] set tenant work area memory(tenant_id=1003, wa_pctg=5, limit=157286400, ret=0) [2025-04-03 14:01:15.107568] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.107612] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.107646] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.107662] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.117852] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.119387] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:311) [18254][T1002_ArcFetche][T1002][YB42AC140005-000631D6BE6CB54C-0-0] [lt=1363] ObArchiveFetcher is running(thread_index=1) [2025-04-03 14:01:15.126237] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=94] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:15.146697] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=4599722598, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:15.146716] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.146786] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=4797790655, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:15.146799] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.146878] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1002}, delta=4810077911, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:15.146889] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.146950] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1003}, delta=4595662906, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:15.146963] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.158774] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=36][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.164377] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [17821][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=77] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b7309dfc2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-04-03 14:01:15.165849] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4809491942, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.165895] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=38] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.171867] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.177903] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.177970] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.177989] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13] re_register_parent reason(palf_id=1001, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660070967313, last_first_register_time_us=1743648246226943, register_time_us=1743660070967313, curr_time_us=1743660075177987) [2025-04-03 14:01:15.178240] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.178280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.178296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.178755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.178784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.179112] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.179168] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660075178482]}) [2025-04-03 14:01:15.179198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.179219] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.179360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.179392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.179764] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660075179188]}) [2025-04-03 14:01:15.179908] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.179931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.180001] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.180710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.180732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.180895] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660075179786]}) [2025-04-03 14:01:15.181293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.181317] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.181341] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660075180914]}) [2025-04-03 14:01:15.181352] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1003, need_refresh=false, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660075180914]}) [2025-04-03 14:01:15.181457] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4677595105, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:15.181502] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.181612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.181631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.181924] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.181963] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=37][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.181980] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.182049] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=57] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.182079] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075182076, replica_locations:[]}) [2025-04-03 14:01:15.182126] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.182124] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=41][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.182141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.182146] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=20][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075179152}) [2025-04-03 14:01:15.182164] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=17][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.182176] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A8-0-0] [lt=11][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.182660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.182698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.182740] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075181669]}) [2025-04-03 14:01:15.182777] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004, need_refresh=false, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075181669]}) [2025-04-03 14:01:15.183133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.183159] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.183325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.183348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.183585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.183610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.184127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.184149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.184576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.184601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.184968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.184993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.185298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.185323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.185490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.185513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.185555] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.185603] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1001, self="172.20.0.5:2882") [2025-04-03 14:01:15.185632] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.196690] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:654) [17572][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=1103] dump dest keepalive data states, addr: "172.20.0.4:2882", last_write_ts_=1743660074796577, last_read_ts_=1743660074996610, in_black_=0 [2025-04-03 14:01:15.196758] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:654) [17572][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=22] dump dest keepalive data states, addr: "172.20.0.3:2882", last_write_ts_=1743660074996650, last_read_ts_=1743660074796593, in_black_=0 [2025-04-03 14:01:15.196777] INFO [RPC.OBRPC] do_client_loop (ob_net_keepalive.cpp:654) [17572][KeepAliveClient][T0][Y0-0000000000000000-0-0] [lt=7] dump dest keepalive data states, addr: "172.20.0.5:2882", last_write_ts_=1743660074796622, last_read_ts_=1743660074996660, in_black_=0 [2025-04-03 14:01:15.207649] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.207697] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=40] re_register_parent reason(palf_id=1002, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660071204208, last_first_register_time_us=1743648247019455, register_time_us=1743660071204208, curr_time_us=1743660075207695) [2025-04-03 14:01:15.207786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.207831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.207991] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208693] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.208942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209244] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.209776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.210083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.210100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.210213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.210228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.210345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.210359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.210486] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="172.20.0.5:2882") [2025-04-03 14:01:15.210515] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.210543] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.210561] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.218105] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.218899] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.218949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=45] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.218967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.219901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=47] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.220806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=46] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.221976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.222157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.222172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.222309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.222324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.222557] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.222716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=157] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.222878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.222894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.223047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.223063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.223305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.223322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.224153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.224175] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.224330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.224345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.224509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.224524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.224720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.224735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.225136] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.225153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.225288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.225304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.227179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.227197] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.227500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.227538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=37] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.227799] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.227675] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 268.00; ) [2025-04-03 14:01:15.227848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=48] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.227976] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=175] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:15.228163] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=68] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:15.228191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.228204] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.228699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.228720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.228866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.228878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229421] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.229443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229566] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FA-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.229840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229959] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.229971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230255] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230374] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.230956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231057] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231539] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.231942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232075] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232285] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232470] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.232933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233097] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.233176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233544] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.233967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.234958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.235064] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.235153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.235260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.235350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.235438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.235526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.235616] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.235762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.246491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.246563] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=12038) [2025-04-03 14:01:15.246582] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=18][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=12050) [2025-04-03 14:01:15.246614] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.246631] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.246660] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=28] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.246680] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.246715] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:582) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=10] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-04-03 14:01:15.246730] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:221) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=308) [2025-04-03 14:01:15.246785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.246938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.246964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247254] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247653] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.247963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248437] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248727] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248852] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.248963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.249053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.249160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.249171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.249350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.249824] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.250047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.250057] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.250151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.250298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.250324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.250482] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4599826018, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:15.250506] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.250676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.250838] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1001}, delta=4797895155, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:15.250862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.250861] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.251005] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.251155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.251192] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1002}, delta=4810182411, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:15.251234] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=38] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.251273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.251513] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1003}, delta=4595767406, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:15.251525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.251532] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.251591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.251747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17817][T1_LSMetaCh][T1][YB42AC140005-000631D6BB6CE2DB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.258854] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.259093] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE375-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.262920] ERROR issue_dba_error (ob_log.cpp:1875) [18260][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4013, file="ob_multi_version_garbage_collector.cpp", line_no=222, info="repeat refresh failed too much time") [2025-04-03 14:01:15.262945] EDIAG [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:222) [18260][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] repeat refresh failed too much time(ret=-4013, this={this:0x2b73313c71a0, last_study_timestamp:1743655034213402, last_refresh_timestamp:1743655214204510, last_reclaim_timestamp:1743659835989550, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:true, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1743654670704737796, v:0}, is_inited:true}, current_timestamp=1743660075024274) BACKTRACE:0x125fd785 0x50b6335 0x51fab28 0x51fa63b 0x51fa57e 0x51fa3a6 0xef83d9e 0xef927e9 0x5209517 0x5206c75 0x520639e 0x7662b60 0x128d17e1 0x128cdbbf 0x2b72da6c4ea5 0x2b72da9d7b0d [2025-04-03 14:01:15.263068] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A9-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.263090] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A9-0-0] [lt=21][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.263510] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A9-0-0] [lt=1] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075263508, replica_locations:[]}) [2025-04-03 14:01:15.263580] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A9-0-0] [lt=66][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.263601] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A9-0-0] [lt=20][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075257359}) [2025-04-03 14:01:15.263613] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A9-0-0] [lt=11][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.263621] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6A9-0-0] [lt=7][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.272239] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.273112] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.273728] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.273793] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.277964] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.277995] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.278030] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.280095] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.281588] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.284289] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.286875] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.286920] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=44][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.286928] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=7] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.287079] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=6] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.287212] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075287210, replica_locations:[]}) [2025-04-03 14:01:15.287241] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=26][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.287248] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=7][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075284332}) [2025-04-03 14:01:15.287256] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=7][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.287260] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AA-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.307734] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.307787] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.307862] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=69][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.307880] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.312726] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18111][T1002_LogUpdate][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.317009] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28466 [2025-04-03 14:01:15.318194] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.345077] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [17788][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.347669] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1002}, delta=4810279030, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:15.347798] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1003}, delta=4595864025, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:15.358980] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.359577] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.359619] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28470 [2025-04-03 14:01:15.359697] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28470 [2025-04-03 14:01:15.359742] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.363223] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:397) [17417][ServerTracerTim][T0][YB42AC140005-000631D6BBECBE95-0-0] [lt=29166] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-04-03 14:01:15.366009] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4809692792, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.366080] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=63] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.368851] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [17520][DumpLSLoc][T0][YB42AC140005-000631D6C53CB9ED-0-0] [lt=131] [LS_LOCATION]dump tenant ls location caches(tenant_id=1, tenant_ls_locations=[{cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}]) [2025-04-03 14:01:15.368916] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [17520][DumpLSLoc][T0][YB42AC140005-000631D6C53CB9ED-0-0] [lt=63] [LS_LOCATION]dump tenant ls location caches(tenant_id=1001, tenant_ls_locations=[{cache_key:{tenant_id:1001, ls_id:{id:1}, cluster_id:1}, renew_time:1743659485212982, replica_locations:[{server:"172.20.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:430}, {server:"172.20.0.4:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-04-03 14:01:15.368953] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [17520][DumpLSLoc][T0][YB42AC140005-000631D6C53CB9ED-0-0] [lt=34] [LS_LOCATION]dump tenant ls location caches(tenant_id=1002, tenant_ls_locations=[{cache_key:{tenant_id:1002, ls_id:{id:1002}, cluster_id:1}, renew_time:1743659139992537, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:171}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1}, cluster_id:1}, renew_time:1743659972553089, replica_locations:[{server:"172.20.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:454}, {server:"172.20.0.4:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1003}, cluster_id:1}, renew_time:1743659139992544, replica_locations:[{server:"172.20.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:104}, {server:"172.20.0.4:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1002, ls_id:{id:1001}, cluster_id:1}, renew_time:1743659139992534, replica_locations:[{server:"172.20.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1292}, {server:"172.20.0.4:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-04-03 14:01:15.369135] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [17520][DumpLSLoc][T0][YB42AC140005-000631D6C53CB9ED-0-0] [lt=181] [LS_LOCATION]dump tenant ls location caches(tenant_id=1003, tenant_ls_locations=[{cache_key:{tenant_id:1003, ls_id:{id:1}, cluster_id:1}, renew_time:1743659485218542, replica_locations:[{server:"172.20.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:620}, {server:"172.20.0.4:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-04-03 14:01:15.369246] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.369180] INFO [SHARE.LOCATION] dump_cache (ob_ls_location_service.cpp:1206) [17520][DumpLSLoc][T0][YB42AC140005-000631D6C53CB9ED-0-0] [lt=42] [LS_LOCATION]dump tenant ls location caches(tenant_id=1004, tenant_ls_locations=[{cache_key:{tenant_id:1004, ls_id:{id:1002}, cluster_id:1}, renew_time:1743659183789028, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:1}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1001}, cluster_id:1}, renew_time:1743659140006352, replica_locations:[{server:"172.20.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:637}, {server:"172.20.0.4:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1}, cluster_id:1}, renew_time:1743659890238781, replica_locations:[{server:"172.20.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:627}, {server:"172.20.0.4:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}, {cache_key:{tenant_id:1004, ls_id:{id:1003}, cluster_id:1}, renew_time:1743659140006360, replica_locations:[{server:"172.20.0.3:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:14}, {server:"172.20.0.4:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.5:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}]}]) [2025-04-03 14:01:15.369287] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=33] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28472 [2025-04-03 14:01:15.369385] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28472 [2025-04-03 14:01:15.369409] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.372037] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.375454] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.375498] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28474 [2025-04-03 14:01:15.375587] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28474 [2025-04-03 14:01:15.375614] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.378070] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.378124] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.378143] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.380176] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.381681] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4677795746, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:15.381759] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=70] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.384846] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.384887] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=33] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28476 [2025-04-03 14:01:15.384978] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28476 [2025-04-03 14:01:15.385006] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.388269] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660075387372]}) [2025-04-03 14:01:15.388988] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660075388313]}) [2025-04-03 14:01:15.389632] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660075389042]}) [2025-04-03 14:01:15.390185] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660075389659]}) [2025-04-03 14:01:15.390735] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.390771] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=34][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.390787] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.390979] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.391094] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=18] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075391092, replica_locations:[]}) [2025-04-03 14:01:15.391190] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=92][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.391210] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=19][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075388236}) [2025-04-03 14:01:15.391229] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=17][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.391240] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AB-0-0] [lt=10][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.391490] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075390561]}) [2025-04-03 14:01:15.404751] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28478 [2025-04-03 14:01:15.407834] INFO [PALF] log_loop_ (log_loop_thread.cpp:175) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] LogLoopThread round_cost_time(us)(round_cost_time=203) [2025-04-03 14:01:15.407907] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=55][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.407946] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.407965] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.407991] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.415844] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:15.415871] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28480 [2025-04-03 14:01:15.415938] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28480 [2025-04-03 14:01:15.415956] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:15.418283] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.431740] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.431801] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=53] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28482 [2025-04-03 14:01:15.431917] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28482 [2025-04-03 14:01:15.431994] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=51] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.450289] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4600026743, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:15.450325] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.450377] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=24] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1001}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-04-03 14:01:15.450466] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.450421] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=39] get_min_unreplayed_log_info(lsn={lsn:3574142328}, scn={val:1743655277355487644, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1743660074780177, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:3574142328}, next_to_submit_scn_:{val:1743655277355487644, v:0}, base_lsn_:{lsn:3489447936}, base_scn_:{val:1743616894738777400, v:0}, iterator_:{iterator_impl:{buf_:0x2b7357205000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:3574142206}, end_lsn:{lsn:3574142328}, read_buf:{buf_len_:253952, buf_:0x2b7357205000}, block_size:67104768, log_storage_:0x2b7378b65bf0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1743655277355487643, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1743655277355487643, v:0}, data_checksum:154833137, flag:0}}, init_mode_version:0, accumulate_checksum:3441221005, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1743655277355487643, v:0}}}}}) [2025-04-03 14:01:15.450498] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=73] get wrs ts(ls_id={id:1001}, delta=4798094800, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:15.450499] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28484 [2025-04-03 14:01:15.450512] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.450587] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28484 [2025-04-03 14:01:15.450604] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.450848] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:0, v:0}, this={ls_id_:{id:1002}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:0, v:0}}) [2025-04-03 14:01:15.450888] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=33] get_min_unreplayed_log_info(lsn={lsn:74011080}, scn={val:1743655265068231481, v:0}, this={ls_id_:{id:1002}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1743660074780191, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:74011080}, next_to_submit_scn_:{val:1743655265068231481, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1743616823359480140, v:0}, iterator_:{iterator_impl:{buf_:0x2b73b4d53000, next_round_pread_size:2121728, curr_read_pos:122, curr_read_buf_start_pos:0, curr_read_buf_end_pos:122, log_storage_:{IteratorStorage:{start_lsn:{lsn:74010958}, end_lsn:{lsn:74011080}, read_buf:{buf_len_:253952, buf_:0x2b73b4d53000}, block_size:67104768, log_storage_:0x2b73ab3c7bf0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1743655265068231480, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1743655265068231480, v:0}, data_checksum:3348837161, flag:0}}, init_mode_version:0, accumulate_checksum:1277512058, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1743655265068231480, v:0}}}}}) [2025-04-03 14:01:15.450958] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=66] get wrs ts(ls_id={id:1002}, delta=4810382056, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:15.450974] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.451319] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=29] get_min_unreplayed_log_info(lsn={lsn:74920229}, scn={val:1743655479483236416, v:0}, this={ls_id_:{id:1003}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1743660074780196, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:3280, ret_code_:0}}, next_to_submit_lsn_:{lsn:74920229}, next_to_submit_scn_:{val:1743655479483236416, v:0}, base_lsn_:{lsn:0}, base_scn_:{val:1743616894738777400, v:0}, iterator_:{iterator_impl:{buf_:0x2b7587e47000, next_round_pread_size:2121728, curr_read_pos:249052, curr_read_buf_start_pos:0, curr_read_buf_end_pos:249052, log_storage_:{IteratorStorage:{start_lsn:{lsn:74671177}, end_lsn:{lsn:74920229}, read_buf:{buf_len_:253952, buf_:0x2b7587e47000}, block_size:67104768, log_storage_:0x2b7372ff7bf0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1743655479483236415, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1743655479483236415, v:0}, data_checksum:2124447694, flag:0}}, init_mode_version:0, accumulate_checksum:2102179633, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1743655479483236415, v:0}}}}}) [2025-04-03 14:01:15.451411] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=88] get wrs ts(ls_id={id:1003}, delta=4595967051, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:15.451427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.459069] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=64][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.460991] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:45) [17414][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0] dump dtl interm result cost(us)(dump_cost=36209, ret=0, interm count=0, dump count=1182) [2025-04-03 14:01:15.469637] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4809795787, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.469698] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=55] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.472108] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.473323] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.473341] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28486 [2025-04-03 14:01:15.473374] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28486 [2025-04-03 14:01:15.473381] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.478158] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.478214] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.478234] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.480249] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.480533] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:60) [17414][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=41] clear dtl interm result cost(us)(clear_cost=19498, ret=0, expire_keys_.count()=0, dump count=1182, interm count=0, clean count=0) [2025-04-03 14:01:15.485990] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.486051] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=52] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28488 [2025-04-03 14:01:15.486135] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28488 [2025-04-03 14:01:15.486161] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.492648] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=0] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=9) [2025-04-03 14:01:15.493239] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660075492323]}) [2025-04-03 14:01:15.493925] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.494204] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=270] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28490 [2025-04-03 14:01:15.494276] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660075493304]}) [2025-04-03 14:01:15.494312] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28490 [2025-04-03 14:01:15.494337] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.495133] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660075494311]}) [2025-04-03 14:01:15.495627] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.495654] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=26][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.495671] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=16] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.495834] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.496004] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=16] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075496002, replica_locations:[]}) [2025-04-03 14:01:15.496071] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=63][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.496082] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=10][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075493212}) [2025-04-03 14:01:15.496091] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=8][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.496101] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AC-0-0] [lt=9][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.496724] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [18600][T1004_LockWaitM][T1004][Y0-0000000000000000-0-0] [lt=13037] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-04-03 14:01:15.496984] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660075495156]}) [2025-04-03 14:01:15.498856] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075497311]}) [2025-04-03 14:01:15.499223] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:15.499266] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.508097] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.509297] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.509360] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.509398] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.510728] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.510777] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.517947] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28494 [2025-04-03 14:01:15.518380] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.520734] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.520862] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28496 [2025-04-03 14:01:15.523493] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.528324] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=79] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:15.547385] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=16] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.547435] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=43] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.547591] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=36] get wrs ts(ls_id={id:1001}, delta=4798191454, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:15.547958] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1002}, delta=4810478710, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:15.559161] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.566170] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=4809892398, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.566264] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=86] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.566873] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:103) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] tx gc loop thread is running(MTL_ID()=1) [2025-04-03 14:01:15.567205] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=29] update calc upper info once(calc_upper_info_={min_start_scn_in_ctx_:{val:1743655261104510952, v:0}, keep_alive_scn_:{val:1743655261104510952, v:0}, update_ts_:1743660075567201}) [2025-04-03 14:01:15.567885] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.567923] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=37][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.567941] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=17] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.568080] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=18] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.568271] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075568268, replica_locations:[]}) [2025-04-03 14:01:15.568333] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=57][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.568355] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=20][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075563365}) [2025-04-03 14:01:15.568386] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=29][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.568404] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AD-0-0] [lt=17][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.569151] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.569193] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28500 [2025-04-03 14:01:15.569265] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28500 [2025-04-03 14:01:15.569294] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.569440] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=34] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616801173178850, v:0}, clog_base_lsn:{lsn:13286744064}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1743618797082332550, v:0}, all_id_meta:{id_meta:[{limited_id:1743655279117594735, latest_log_ts:{val:1743655262469466896, v:0}}, {limited_id:49000001, latest_log_ts:{val:1743605935907428857, v:0}}, {limited_id:360000001, latest_log_ts:{val:1743640357777174079, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:159, palf_env_:0x2b730a3fc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:7936, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:15.572211] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.578249] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.578305] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.578324] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.578717] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.578767] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=42] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28502 [2025-04-03 14:01:15.578863] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28502 [2025-04-03 14:01:15.578884] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.580347] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.582046] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4677995618, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:15.582112] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=43] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.584835] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.584861] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28504 [2025-04-03 14:01:15.584902] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28504 [2025-04-03 14:01:15.584914] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.593901] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=42] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.593925] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28506 [2025-04-03 14:01:15.593973] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28506 [2025-04-03 14:01:15.593981] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.596410] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [18288][T1003_Occam][T1003][YB42AC140005-000631D6BF9CDA69-0-0] [lt=36] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-04-03 14:01:15.596518] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [18288][T1003_Occam][T1003][YB42AC140005-000631D6BF9CDA6B-0-0] [lt=41] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-04-03 14:01:15.601066] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.601090] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28508 [2025-04-03 14:01:15.601140] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28508 [2025-04-03 14:01:15.601149] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.603753] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660075600752]}) [2025-04-03 14:01:15.604511] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.604935] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660075603804]}) [2025-04-03 14:01:15.607506] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660075604991]}) [2025-04-03 14:01:15.608159] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.608213] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.608236] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.608255] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.610142] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660075607545]}) [2025-04-03 14:01:15.611067] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075610481]}) [2025-04-03 14:01:15.612518] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.612559] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=39][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.612573] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.612764] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.613039] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=24] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075613037, replica_locations:[]}) [2025-04-03 14:01:15.613108] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=65][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.613126] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=17][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075603721}) [2025-04-03 14:01:15.613143] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=16][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.613155] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AE-0-0] [lt=10][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.618504] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.627024] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17558][RpcIO][T0][Y0-0000000000000000-0-0] [lt=23] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.639192] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17808][T1_RFLWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.639882] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.640242] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=65] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:15.640610] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.640671] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=53] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28510 [2025-04-03 14:01:15.640748] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28510 [2025-04-03 14:01:15.640782] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.650562] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.650610] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=39] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28512 [2025-04-03 14:01:15.650691] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28512 [2025-04-03 14:01:15.651805] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4600227564, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:15.651876] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=64] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.652042] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1001}, delta=4798295621, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:15.652088] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.652213] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=36] get wrs ts(ls_id={id:1002}, delta=4810582877, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:15.659239] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.666041] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.666056] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28514 [2025-04-03 14:01:15.666090] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28514 [2025-04-03 14:01:15.666097] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.666466] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.666967] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:110) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=131] try gc retain ctx [2025-04-03 14:01:15.669302] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.669628] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4809995501, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.669671] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.672264] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=57][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.675009] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.675039] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28516 [2025-04-03 14:01:15.675072] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28516 [2025-04-03 14:01:15.675080] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.678224] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.678572] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.678636] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.678656] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.679074] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [17830][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=28] ====== traversal_flush timer task ====== [2025-04-03 14:01:15.679240] INFO [MDS] flush (mds_table_mgr.cpp:172) [17830][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=23] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.680439] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.683364] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.683456] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4678097484, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:15.685717] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.685742] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28520 [2025-04-03 14:01:15.685802] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28520 [2025-04-03 14:01:15.685831] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.693044] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=8) [2025-04-03 14:01:15.694065] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.694121] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=48] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28522 [2025-04-03 14:01:15.694188] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28522 [2025-04-03 14:01:15.694203] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.704928] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=17][errcode=-4012] acquire global snapshot fail(ret=-4012, gts_ahead=0, expire_ts=1743660075703884, now=[mts=1743660074760120], now0=[mts=1743660074760120], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:15.704997] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=63][errcode=-4012] acquire global snapshot fail(ret=-4012, tx={this:0x2b73a4335d80, tx_id:{txid:76175011}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660074757183, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:15.705106] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=101][errcode=-4012] fail to get snapshot(ret=-4012, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73961321a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a4335d80}) [2025-04-03 14:01:15.705158] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=49][errcode=-4012] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4012, session_id=1, *tx_desc={this:0x2b73a4335d80, tx_id:{txid:76175011}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660074757183, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:15.705218] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4012] fail to start stmt(ret=-4012, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:15.705240] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=19][errcode=-4012] fail start stmt(ret=-4012) [2025-04-03 14:01:15.705251] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4012] execute plan failed(ret=-4012) [2025-04-03 14:01:15.705262] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=8][errcode=-4012] open result set failed(ret=-4012) [2025-04-03 14:01:15.705292] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=27][errcode=-4012] result set open failed(ret=-4012, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:15.705334] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=39][errcode=-4012] execute failed(ret=-4012, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=23, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:15.705355] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=18][errcode=-4012] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:23, local_retry_times:23, err_:-4012, err_:"OB_TIMEOUT", retry_type:0, client_ret:-4012}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:15.705408] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=36][errcode=-4012] result set close failed(ret=-4012) [2025-04-03 14:01:15.705419] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=10][errcode=-4012] result set close failed(ret=-4012) [2025-04-03 14:01:15.705427] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=7][errcode=-4012] failed to close result(close_ret=-4012, ret=-4012) [2025-04-03 14:01:15.705628] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=44] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.708313] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.708362] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.708390] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.708409] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.711369] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34D-0-0] [lt=1][errcode=-4012] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4012, ret=-4012) [2025-04-03 14:01:15.711421] INFO [SERVER] process_final (ob_inner_sql_connection.cpp:662) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=47] slow inner sql(last_ret=-4012, sql={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, process_time=29937526) [2025-04-03 14:01:15.711450] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=28][errcode=-4012] failed to process final(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, aret=-4012, ret=-4012) [2025-04-03 14:01:15.711466] WDIAG [SERVER] execute_write_inner (ob_inner_sql_connection.cpp:1451) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4012] execute sql failed(ret=-4012, tenant_id=1003, sql=INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = '') [2025-04-03 14:01:15.711500] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4012] retry_while_no_tenant_resource failed(ret=-4012, tenant_id=1003) [2025-04-03 14:01:15.711513] WDIAG [SERVER] execute_write (ob_inner_sql_connection.cpp:1406) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=12][errcode=-4012] execute_write failed(ret=-4012, tenant_id=1003, sql=INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = '', is_user_sql=false) [2025-04-03 14:01:15.711527] WDIAG [COMMON.MYSQLP] write (ob_mysql_proxy.cpp:156) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4012] execute sql failed(ret=-4012, conn=0x2b7396132050, start=1743660045773402, sql=INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = '') [2025-04-03 14:01:15.711630] WDIAG [SHARE] exec_insert_update (ob_dml_sql_splicer.cpp:1283) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=36][errcode=-4012] execute sql failed(ret=-4012, sql=INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = '') [2025-04-03 14:01:15.711645] WDIAG [SHARE.PT] update_replica_ (ob_persistent_ls_table.cpp:548) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=13][errcode=-4012] execute update failed(ret=-4012, ret="OB_TIMEOUT", replica={modify_time_us:1743660045773360, create_time_us:0, tenant_id:1004, ls_id:{id:1003}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648246150910}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1006, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:15.711718] WDIAG [SHARE.PT] update (ob_persistent_ls_table.cpp:405) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=50][errcode=-4012] update follower replica failed(ret=-4012, ret="OB_TIMEOUT", new_replica={modify_time_us:1743660045773360, create_time_us:0, tenant_id:1004, ls_id:{id:1003}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648246150910}], replica_type:0, proposal_id:0, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1006, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:15.711763] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=41][errcode=-4012] update replica failed(ret=-4012, ret="OB_TIMEOUT", replica={modify_time_us:0, create_time_us:0, tenant_id:1004, ls_id:{id:1003}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648246150910}], replica_type:0, proposal_id:17, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1006, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:15.711840] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=75] update ls replica(ret=-4012, ret="OB_TIMEOUT", time_used=29938480, replica={modify_time_us:0, create_time_us:0, tenant_id:1004, ls_id:{id:1003}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648246150910}], replica_type:0, proposal_id:17, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1006, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:15.711877] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17486][LSUserTblUp0][T0][YB42AC140005-000631D6BD2CC34D-0-0] [lt=36][errcode=-4012] fail to update replica(ret=-4012, ret="OB_TIMEOUT", tenant_id=1004, ls_id={id:1003}, replica={modify_time_us:0, create_time_us:0, tenant_id:1004, ls_id:{id:1003}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648246150910}], replica_type:0, proposal_id:17, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1006, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:15.712764] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660075711686]}) [2025-04-03 14:01:15.713379] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660075712838]}) [2025-04-03 14:01:15.713975] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660075713419]}) [2025-04-03 14:01:15.714445] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660075714006]}) [2025-04-03 14:01:15.715327] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.715391] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=62][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.715442] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=49] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.715549] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.715616] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=19] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075715614, replica_locations:[]}) [2025-04-03 14:01:15.715640] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075714853]}) [2025-04-03 14:01:15.715655] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=35][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.715671] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=14][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075712735}) [2025-04-03 14:01:15.715691] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=19][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.715705] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6AF-0-0] [lt=13][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.716402] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=42] ====== check clog disk timer task ====== [2025-04-03 14:01:15.716450] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=35] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=315) [2025-04-03 14:01:15.716994] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=13] cannot_recycle_log_size statistics(cannot_recycle_log_size=129898528, threshold=483183820, need_update_checkpoint_scn=false) [2025-04-03 14:01:15.717595] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:15.717637] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28524 [2025-04-03 14:01:15.717717] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28524 [2025-04-03 14:01:15.717756] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:15.718593] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.724771] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:15.748461] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1002}, {id:1003}]) [2025-04-03 14:01:15.750727] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1003}, delta=4596267307, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:15.755307] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:121) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=44269] new mysql sessid created(conn.sessid_=3222021278, support_ssl=false) [2025-04-03 14:01:15.755486] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:141) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=89] sm conn init succ(conn.sessid_=3222021278, sess.client_addr_="127.0.0.1:34902") [2025-04-03 14:01:15.755516] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:942) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x2b731cffe710, session_id:3222021278, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, fd:93, err:0, last_decode_time:0, last_write_time:1743660075755479, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-04-03 14:01:15.757038] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1] ====== tenant freeze timer task ====== [2025-04-03 14:01:15.757476] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.757667] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.757705] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.757736] ERROR issue_dba_error (ob_log.cpp:1875) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4013, file="ob_mysql_protocol_processor.cpp", line_no=135, info="fail to alloc memory for OMPKHandshakeResponse") [2025-04-03 14:01:15.757756] EDIAG [RPC.OBMYSQL] decode_hsr_body (ob_mysql_protocol_processor.cpp:135) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] fail to alloc memory for OMPKHandshakeResponse(alloc_size=588, ret=-4013) BACKTRACE:0x125fd785 0x50b6335 0x51ad5ed 0x51ad0af 0x519a224 0x51acf53 0x1339477e 0x4d023dd 0x4cfef86 0x128cdbed 0x2b72da6c4ea5 0x2b72da9d7b0d [2025-04-03 14:01:15.758368] ERROR issue_dba_error (ob_log.cpp:1875) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4013, file="ob_mysql_protocol_processor.cpp", line_no=135, info="fail to alloc memory for OMPKHandshakeResponse") [2025-04-03 14:01:15.758395] EDIAG [RPC.OBMYSQL] decode_hsr_body (ob_mysql_protocol_processor.cpp:135) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] fail to alloc memory for OMPKHandshakeResponse(alloc_size=588, ret=-4013) BACKTRACE:0x125fd785 0x50b6335 0x51ad5ed 0x51ad0af 0x519a224 0x51acf53 0x1339477e 0x4d023dd 0x4d0147d 0x4cff429 0x128cdbed 0x2b72da6c4ea5 0x2b72da9d7b0d [2025-04-03 14:01:15.758418] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:748) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4015] close sql sock by user req(*s={this:0x2b731cffe710, session_id:3222021278, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, fd:93, err:5, last_decode_time:0, last_write_time:1743660075755479, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-04-03 14:01:15.758469] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [17574][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=0] connection close(sessid=3222021278, proxy_sessid=0, tenant_id=0, server_id=3, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:0], disconnect_state=0) [2025-04-03 14:01:15.759255] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.759429] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.761298] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [17546][Blacklist][T0][Y0-0000000000000000-0-0] [lt=55] blacklist_loop exec finished(cost_time=46, is_enabled=true, send_cnt=0) [2025-04-03 14:01:15.763232] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:13578, tid:17513}]) [2025-04-03 14:01:15.764840] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=12][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.764874] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=32][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.766277] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=36] get wrs ts(ls_id={id:1}, delta=4810093037, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.766321] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.767828] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.767889] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=58][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.767905] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.768168] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.768395] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=24] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075768392, replica_locations:[]}) [2025-04-03 14:01:15.768456] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=57][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.768478] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=21][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075762824}) [2025-04-03 14:01:15.768497] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=17][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.768510] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B0-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.772343] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.776527] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:15.776593] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=57] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28526 [2025-04-03 14:01:15.776668] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28526 [2025-04-03 14:01:15.776708] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:15.778860] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.778897] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.779104] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.779131] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.779527] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.779553] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.779878] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.779914] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.780529] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.780729] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.780784] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.781486] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.781535] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:15.781554] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.781644] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=2][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:15.781665] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=20][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.781959] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:15.781979] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.782069] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:15.782081] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.782169] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:15.782179] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.782299] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:15.782338] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=38][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.782500] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:15.782520] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.782590] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1004, wash_size=6291456) [2025-04-03 14:01:15.782635] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18507][T1004_TeRec][T1004][YB42AC140005-000631D6C02CE3B6-0-0] [lt=43][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.782711] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4678197113, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:15.782757] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=38] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.783208] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.783257] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.783952] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.783983] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.784230] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.784281] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.784293] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.784619] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.784646] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=26][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.785069] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.785103] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=33][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.785179] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.785193] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.785376] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.785401] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.785657] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.785682] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.785787] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.785802] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.786009] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.786046] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=36][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.786285] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.786327] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.786565] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.786593] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=27][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.786837] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.786857] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.787186] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.787221] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.787449] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.787489] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=39][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.787708] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.787734] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=25][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.787917] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.787938] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.788055] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.788065] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.788289] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.788308] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.788482] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.788505] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=22][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.788711] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.788730] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.788805] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.788842] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.789100] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.789130] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=29][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.789352] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.789366] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.789603] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.789618] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.789771] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.789786] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.790043] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.790058] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.790293] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.790308] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.790417] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.790432] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.790679] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.790715] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=35][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.790973] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.790988] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.791208] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:15.791223] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.791555] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.791571] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.791863] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:15.791878] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:15.793177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.793260] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4389] destruct(*this=time guard 'poc_rpc_post' cost too much time, used=14601) [2025-04-03 14:01:15.793275] WDIAG [LIB] ~ObTimeGuard (utility.h:857) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4389] destruct(*this=time guard 'rpc post' cost too much time, used=14612) [2025-04-03 14:01:15.793296] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.793334] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.793348] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.794223] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=30] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.794577] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18390][T1003_DupTbLeas][T1][YB42AC140005-000631D6C28CB928-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.808398] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.808447] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.808468] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.808486] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.809970] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18235][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.810880] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.818737] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.819584] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660075818338]}) [2025-04-03 14:01:15.820980] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.821003] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=22][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.821011] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=7] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.821003] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075819989]}) [2025-04-03 14:01:15.821149] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=19] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.821268] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=16] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075821267, replica_locations:[]}) [2025-04-03 14:01:15.821304] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=34][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.821312] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=8][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075817221}) [2025-04-03 14:01:15.821339] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=26][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.821344] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B1-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.847240] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [17564][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=30] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:15.849849] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:15.851164] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=51] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:15.851344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.852052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.852647] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4600428441, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:15.853386] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:15.857554] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:15.857589] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28534 [2025-04-03 14:01:15.857667] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28534 [2025-04-03 14:01:15.857683] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:15.859495] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.866543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.866580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.866799] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:15.866862] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=55] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28536 [2025-04-03 14:01:15.866897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.866913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.866974] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=48] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28536 [2025-04-03 14:01:15.866993] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:15.867187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.867211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869009] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=93] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.869917] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.870175] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.870203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.870497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.870522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.870766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.870793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.870999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871060] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=60] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871387] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=45] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.871828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=41] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.872305] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4810198320, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.872354] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.872426] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=114][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.875743] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:15.875780] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=28] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28538 [2025-04-03 14:01:15.875876] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28538 [2025-04-03 14:01:15.875893] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:15.878734] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.878770] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.878784] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.880620] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.882786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.882823] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.882919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.882931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883060] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883251] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883929] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.883943] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.884916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.885053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.885068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.885157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.885169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.885306] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4678298994, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:15.885335] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.885965] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:15.886036] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28540 [2025-04-03 14:01:15.886127] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28540 [2025-04-03 14:01:15.886152] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:15.893179] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=7) [2025-04-03 14:01:15.908491] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.908548] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.908567] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.908623] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.917330] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:15.917348] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28542 [2025-04-03 14:01:15.917382] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28542 [2025-04-03 14:01:15.917388] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:15.918819] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=43] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"172.20.0.5:2882", switch_to_leader_ts:-1}) [2025-04-03 14:01:15.918847] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:15.922389] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660075921754]}) [2025-04-03 14:01:15.923039] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660075922430]}) [2025-04-03 14:01:15.923648] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660075923054]}) [2025-04-03 14:01:15.924416] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660075923681]}) [2025-04-03 14:01:15.925335] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660075924663]}) [2025-04-03 14:01:15.929197] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.929270] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=71][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.929291] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=19] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.929499] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=20] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.929846] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=21] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075929843, replica_locations:[]}) [2025-04-03 14:01:15.929905] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=54][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:15.929927] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=20][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660075922372}) [2025-04-03 14:01:15.929947] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=18][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:15.929960] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B2-0-0] [lt=12][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:15.930971] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=97] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:15.946710] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C11-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.947459] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4600523793, log_service_ts=1743655475423544, min_tx_service_ts=4611686018427387, timestamp={val:1743655475423544390, v:0}) [2025-04-03 14:01:15.947496] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655471770425554, v:0}", last_lsn={lsn:4316151456}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.947550] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1001}, delta=4798591850, log_service_ts=1743655277355487, min_tx_service_ts=4611686018427387, timestamp={val:1743655277355487643, v:0}) [2025-04-03 14:01:15.947560] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655276442494259, v:0}", last_lsn={lsn:3574141183}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.947617] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1002}, delta=4810879106, log_service_ts=1743655265068231, min_tx_service_ts=4611686018427387, timestamp={val:1743655265068231480, v:0}) [2025-04-03 14:01:15.947633] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1002}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655263640007290, v:0}", last_lsn={lsn:74009437}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.947673] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=9] get_max_applied_scn(scn={val:1743655258634689680, v:0}, this={ls_id_:{id:1003}, role_:2, proposal_id_:13, palf_committed_end_lsn_:{lsn:74671177}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:1743655258634689680, v:0}}) [2025-04-03 14:01:15.947696] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1003}, delta=4596464101, log_service_ts=1743655479483236, min_tx_service_ts=4611686018427387, timestamp={val:1743655479483236415, v:0}) [2025-04-03 14:01:15.947703] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=5] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1004, LS_ID={id:1003}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655476744925757, v:0}", last_lsn={lsn:74916766}, last_gts={val:1743659145871421653, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.959537] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=30] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660071954353, last_first_register_time_us=1743648246543257, register_time_us=1743660071954353, curr_time_us=1743660075959535) [2025-04-03 14:01:15.959751] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=37][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.959780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.959822] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=41] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.960166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.960190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.960463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.960498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.960729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.960745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.960941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.960960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.961210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.961228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.961458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.961474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.962078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.962094] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.962549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.962564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.963193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.963226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.963632] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.963650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.963712] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:3985, tid:18702}]) [2025-04-03 14:01:15.964279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.964330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=49] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.964656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.964674] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.965191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.965217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.965669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.965694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.966177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.966201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.966487] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=4810292462, log_service_ts=1743655265673135, min_tx_service_ts=4611686018427387, timestamp={val:1743655265673135133, v:0}) [2025-04-03 14:01:15.966531] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1003, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655262875017342, v:0}", last_lsn={lsn:23016129045}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.966699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.966721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.967169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.967190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.967334] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.967359] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=23][errcode=-4013] oops, alloc failed, tenant_id=1003, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=1073741800, tenant_hold=441917440, tenant_limit=1073741824 [2025-04-03 14:01:15.967377] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_LOG_REGISTER_PARENT_REQ, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.967393] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:15.967407] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] LogRpc post_request failed(ret=-4013, palf_id_=1, req={child:{server:"172.20.0.5:2882", region:"default_region", register_time_us:1743660075959581, keepalive_ts:-1}, is_to_leader:true}, server="172.20.0.3:2882") [2025-04-03 14:01:15.967451] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=41] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.20.0.5:2882") [2025-04-03 14:01:15.967473] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.969304] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=32] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1}, {id:1001}]) [2025-04-03 14:01:15.972511] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=37][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.972700] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:15.972731] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=28][errcode=-4013] oops, alloc failed, tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=9223372036854775807, tenant_hold=8096894976, tenant_limit=9223372036854775807 [2025-04-03 14:01:15.972757] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=18][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.972773] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=15][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:15.972836] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=59][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.972866] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=29][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.4:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.972885] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.973221] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=21][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:15.973236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973299] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.973309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973313] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973506] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973651] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=89] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973917] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.973992] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.973997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974001] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974065] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974481] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.974618] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18431][T1003_HAService][T1003][Y0-0000000000000000-0-0] [lt=0] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:15.975560] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.975588] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=27][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:15.975605] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=13][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.975630] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=24][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.3:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.977293] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.977316] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:15.977330] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=11][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.977371] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=40][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.5:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:15.977443] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.977466] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=22][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:15.977481] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:15.977508] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:15.977535] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660075977533, replica_locations:[]}) [2025-04-03 14:01:15.978706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.978729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.978849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.978862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.978975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.978988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979066] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979077] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979646] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.979757] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.979786] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.979803] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.980655] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:15.982335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.982360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.982546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.982562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.982934] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=4678397209, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:15.982972] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:15.983216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.983234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.983725] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.983906] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.983929] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=21][errcode=-4013] rpc memory pool alloc memory failed(sz=1333, alloc_sz=16256) [2025-04-03 14:01:15.983975] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=44][errcode=-4013] alloc buffer fail(payload_sz=1149) [2025-04-03 14:01:15.983990] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=13][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:15.984009] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1, svr="172.20.0.4:2882", timeout=998247, timeout_timestamp=1743660076977580) [2025-04-03 14:01:15.984047] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=36][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:15.984062] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=11][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:15.984074] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:15.984088] WDIAG [SQL] open (ob_result_set.cpp:158) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=11][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:15.984102] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=11][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:15.984113] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:15.984132] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=17][errcode=-4013] execute failed(ret=-4013, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:15.984152] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=17][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:15.984195] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=31][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:15.984226] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=30][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:15.984237] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:15.984261] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=13][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:15.984279] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=15][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:15.984315] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=33][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.984329] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=12][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1) [2025-04-03 14:01:15.984341] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=11][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1) [2025-04-03 14:01:15.984373] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=12][errcode=-4013] query failed(ret=-4013, conn=0x2b73a2330050, start=1743660075978605, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.984393] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=19][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:15.984406] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.984488] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=13][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001) [2025-04-03 14:01:15.984543] WDIAG [SHARE.LOCATION] renew_location_for_tenant (ob_ls_location_service.cpp:806) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=53][errcode=-4013] fail to get all ls info(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, ls_infos=[]) [2025-04-03 14:01:15.984562] WDIAG [SHARE.LOCATION] renew_all_ls_locations (ob_ls_location_service.cpp:737) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=17][errcode=-4013] renew cache for tenant failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001) [2025-04-03 14:01:15.984637] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.985357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.985372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.985740] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.986029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986046] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.986999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.987141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.987289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.987648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.987912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.987940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.988136] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.988213] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.988225] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=11][errcode=-4013] rpc memory pool alloc memory failed(sz=1301, alloc_sz=16256) [2025-04-03 14:01:15.988234] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=8][errcode=-4013] alloc buffer fail(payload_sz=1117) [2025-04-03 14:01:15.988243] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=8][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:15.988257] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=997893, timeout_timestamp=1743660076984572) [2025-04-03 14:01:15.988282] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=24][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:15.988292] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=8][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:15.988300] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:15.988308] WDIAG [SQL] open (ob_result_set.cpp:158) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:15.988315] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:15.988323] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:15.988461] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=137][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:15.988476] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=13][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:15.988504] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=20][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:15.988511] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:15.988518] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:15.988559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.988766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.988932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.989097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.989435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.989574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.989740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.989891] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.990097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.990253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.990579] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.990730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.990884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.991046] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.991213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.991362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.991561] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.991695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.991950] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:15.991964] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7A-0-0] [lt=12][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:15.991977] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=11][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:15.991987] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=8][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.991997] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=9][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:15.992004] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:15.992024] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=8][errcode=-4013] query failed(ret=-4013, conn=0x2b7376644050, start=1743660075985302, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.992037] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=11][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:15.992047] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.992116] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=22][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:15.992126] WDIAG [SHARE.LOCATION] renew_location_for_tenant (ob_ls_location_service.cpp:806) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=9][errcode=-4013] fail to get all ls info(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, ls_infos=[]) [2025-04-03 14:01:15.992137] WDIAG [SHARE.LOCATION] renew_all_ls_locations (ob_ls_location_service.cpp:737) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10][errcode=-4013] renew cache for tenant failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:15.992852] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.992867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993154] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993804] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.993987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.994138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.994454] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.994464] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=9][errcode=-4013] rpc memory pool alloc memory failed(sz=1333, alloc_sz=16256) [2025-04-03 14:01:15.994470] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1149) [2025-04-03 14:01:15.994476] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:15.994484] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1, svr="172.20.0.4:2882", timeout=998726, timeout_timestamp=1743660076992145) [2025-04-03 14:01:15.994493] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=8][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:15.994499] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:15.994505] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:15.994511] WDIAG [SQL] open (ob_result_set.cpp:158) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:15.994527] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=15][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:15.994532] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1003 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:15.994540] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] execute failed(ret=-4013, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1003 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:15.994548] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:15.994566] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=13][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:15.994572] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:15.994577] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=4][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:15.994587] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1003 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:15.994595] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1003 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:15.994601] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1003 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.994617] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=15][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1) [2025-04-03 14:01:15.994622] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1) [2025-04-03 14:01:15.994628] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=5][errcode=-4013] query failed(ret=-4013, conn=0x2b73937da050, start=1743660075992777, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1003 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.994636] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=8][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:15.994642] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=4][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1003 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:15.994677] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1003) [2025-04-03 14:01:15.994683] WDIAG [SHARE.LOCATION] renew_location_for_tenant (ob_ls_location_service.cpp:806) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] fail to get all ls info(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1003, ls_infos=[]) [2025-04-03 14:01:15.994690] WDIAG [SHARE.LOCATION] renew_all_ls_locations (ob_ls_location_service.cpp:737) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7A-0-0] [lt=7][errcode=-4013] renew cache for tenant failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1003) [2025-04-03 14:01:15.995177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.995299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.995763] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.995941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.996081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.996256] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.996265] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=glibc_malloc, ctx_id=7, prio=0, nbyte=32832) [2025-04-03 14:01:15.996432] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.996463] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:15.996473] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=9][errcode=-4013] rpc memory pool alloc memory failed(sz=1335, alloc_sz=16256) [2025-04-03 14:01:15.996479] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1151) [2025-04-03 14:01:15.996569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.996739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.996899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.997231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.997400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.997548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.997716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.998063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.998208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.998378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:15.998529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.008567] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.008601] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.008613] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.008625] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.018834] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=7] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"172.20.0.5:2882", switch_to_leader_ts:-1}) [2025-04-03 14:01:16.018850] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.023850] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.023869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.023875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024229] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024433] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.024903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.025106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.025114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.025289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.025301] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.025435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.025444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.025601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.025611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026060] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026110] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076025747]}) [2025-04-03 14:01:16.026224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026377] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026543] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076026130]}) [2025-04-03 14:01:16.026722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.026777] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076026557]}) [2025-04-03 14:01:16.026802] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=251, get_gts_with_stc_cnt=98, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-04-03 14:01:16.027006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.027021] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.027059] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076026820]}) [2025-04-03 14:01:16.027070] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1003, gts_rpc_cnt=0, get_gts_cache_cnt=155, get_gts_with_stc_cnt=8216, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-04-03 14:01:16.027434] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:16.027453] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:16.027458] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=5] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:16.027469] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=7] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:16.027484] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660076027482, replica_locations:[]}) [2025-04-03 14:01:16.027505] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1008) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=19][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-04-03 14:01:16.027518] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=12][errcode=-4721] fail to renew location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", task={cluster_id:1, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660076026097}) [2025-04-03 14:01:16.027525] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=6][errcode=-4721] fail to batch process task(ret=-4721) [2025-04-03 14:01:16.027529] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6B3-0-0] [lt=4][errcode=-4721] fail to batch execute task(ret=-4721, tasks.count()=1) [2025-04-03 14:01:16.027916] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660076027252]}) [2025-04-03 14:01:16.029056] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=19][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:16.029114] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=56][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660074831773], now0=[mts=1743660074831773], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:16.029144] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=27][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719468}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660074830992, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:16.029212] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=62][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:16.029246] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=32][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719468}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660074830992, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:16.029284] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=36] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f7dce008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660074829917, tx_desc={this:0x2b7399219100, tx_id:{txid:47719468}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660074830992, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660054039470, use_das=false, nested_level=0, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}, plan=0x2b73976ba050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:16.029374] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=87][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:16.029410] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=32][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:16.029421] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=9][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:16.029430] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:16.029440] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:16.029455] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=13][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=17, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:16.029487] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=28] will sleep(sleep_us=17000, remain_us=8009975, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=17, v.err_=-4283, timeout_timestamp=1743660084039460) [2025-04-03 14:01:16.029524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.029530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.029657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.029668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.029911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.030117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.030494] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18713][TimezoneMgr][T1][YB42AC140005-000631D6C2ECE1F6-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.031321] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=4][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.031660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.031833] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.032116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.032349] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.032597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.032753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.033572] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.033750] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.033918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.034386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.034868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.035542] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.035830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.035943] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.036552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.036662] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.036831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.037002] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.037147] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.037482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.038007] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.038135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.038328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.039056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.039436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.039916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.040085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.040409] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.041064] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.041405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1001][YB42AC140005-000631D6C2ECE1F7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.041576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.041629] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.041878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.042091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.042252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.042695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.042853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.043531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.043836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.044022] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.044321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.044806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.045131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.045463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.045786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.046129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.046432] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.046581] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:17, local_retry_times:17, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:16.046651] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=47][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:16.046672] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:16.046683] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:16.046736] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=12][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:16.046770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.046960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.047158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.047161] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=25][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.047244] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.047407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.047571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.047896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.048059] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.048224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.048420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.048602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.048719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1002][YB42AC140005-000631D6C2ECE1F8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.048881] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.049234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.049694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.050041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.050147] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.050193] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:16.050358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.050530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.050697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.050831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.051003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.051157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.051327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.051486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.051596] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18619][T1004_TenantWea][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.051831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.051993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.052129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.052316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.052468] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.052595] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.052670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.052951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.053287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.053436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.053615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.053789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.054436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.054770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.055071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.055563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.055745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.055902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.056234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.056368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.056735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.057061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.057215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.057419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.057565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.057594] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.058455] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.058463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.058737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1003][YB42AC140005-000631D6C2ECE1F9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.059608] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.060589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.060603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.060716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.060728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.060943] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.060951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.061183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.061192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.061363] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.061374] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.061510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.061518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.061627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.063433] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.063444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.063537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.063876] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:16.063905] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.063935] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=49][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:8342, tid:17422}]) [2025-04-03 14:01:16.063992] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:16.064002] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.064066] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:16.064150] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18713][TimezoneMgr][T1004][YB42AC140005-000631D6C2ECE1FA-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:16.066562] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=28][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.066615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=50] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.066628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.066748] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.066768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.066776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.066877] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.066897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.066918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.067081] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.067106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.067113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.067268] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.067281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.067300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.067417] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.067430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.067438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.067622] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.067652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.069902] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.069921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.069947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.070269] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.070288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.070297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.070536] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.070566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.070574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.070745] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.070770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.071084] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.071101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.071443] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.071473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.071953] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.071970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.072163] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.072190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.072579] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17919][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=58][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.076203] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.076254] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.076266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.076277] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=10][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.076441] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.076463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.076483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.076753] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.076775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.076783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.077067] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.077106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.077114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.077399] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.077421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.077549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=127] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.077930] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.077947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.077956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.078491] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.078525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.078535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.078891] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.079161] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.079201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.079640] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.079656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.079767] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=110] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.079836] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.080091] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.080106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.080266] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.080471] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.080585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=112] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.080725] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.080728] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.081005] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.081035] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.081302] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.081329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.081345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.081805] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.081846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.082046] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.082063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.081529] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.082280] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.082304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.082519] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.082546] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.082745] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.082927] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.083056] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.083082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.083177] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.083287] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.083302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.083340] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.083477] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.083521] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.083546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.083824] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.083881] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=54] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.083989] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.084104] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.084127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.084251] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.084306] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.084521] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.084616] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.084662] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.084696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.084828] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.084980] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.085005] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.085026] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.085116] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18110][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:16.085270] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.085309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.085419] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.085583] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.085596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.085459] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.085849] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.085867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.085912] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.086234] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.086245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.086409] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.086471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.086558] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.086675] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.086925] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.087385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=701] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.087644] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.087683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=297] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.087688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=41] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.087732] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.087781] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.087797] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.087865] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.087899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.087927] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088004] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.087977] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088137] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088199] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.088257] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088302] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088310] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.088349] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088393] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.088388] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088448] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088444] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.088527] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.088622] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088665] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088702] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088707] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.088753] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088830] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.088888] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088935] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.088990] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089023] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.089051] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089105] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089155] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.089210] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089257] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089281] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.089303] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089356] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089410] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089452] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.089497] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.089548] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089632] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089707] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089760] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.089776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.089790] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.089801] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089897] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.089982] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.090043] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.090107] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.090116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=8] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.090148] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.090190] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.090279] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.090321] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:16.090343] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:16.090412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.090558] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.090740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.090909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.091053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.091220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AD6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.091722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.091961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.092281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.092377] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.092675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.092848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.093067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.093165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.093283] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=42] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.093510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.093643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.093860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.094011] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.094153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.094307] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.094489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.094658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.094804] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.094983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.095138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.095281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.095435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.095790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.095953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.096092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.096265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.096437] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.096574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.096777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.096930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.097077] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.097226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.097389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.097713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.097899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.098048] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.098209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.098377] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.098571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.098709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.098882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.099070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.099187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.099365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.099517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.099669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.099844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.099997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.100163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.100516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.100680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.100838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.100994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.101160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.101327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.101476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.101795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.101963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.102123] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.102274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.102462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.102612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.104971] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C1D-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.105004] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C1D-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=369, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.105832] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=573] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:16.106422] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=57] Wash time detail, (compute_wash_size_time=265, refresh_score_time=108, wash_time=35) [2025-04-03 14:01:16.108633] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.108651] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.108660] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.108670] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.113736] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=7281, clean_start_pos=3082786, clean_num=62914) [2025-04-03 14:01:16.115403] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C1E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.115433] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C1E-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=314, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.118920] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.123010] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [18023][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=39] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b731e9e82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-04-03 14:01:16.125945] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C1F-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.125990] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C1F-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=446, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.129767] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076128268]}) [2025-04-03 14:01:16.130905] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076129838]}) [2025-04-03 14:01:16.131495] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076130952]}) [2025-04-03 14:01:16.132084] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076131528]}) [2025-04-03 14:01:16.133145] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660076132448]}) [2025-04-03 14:01:16.136154] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=53] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136888] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.136913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137073] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137212] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137395] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137588] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C20-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.137638] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C20-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1354, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.137754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.137983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138009] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138337] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.138982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=56] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139852] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.139991] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140959] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.140979] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141772] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=86] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.141997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=46] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142229] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.142995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143038] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=41] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143629] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.143992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144033] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144859] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144877] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.144997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145130] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145433] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145763] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145861] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.145975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=135] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146197] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146861] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.146969] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147246] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147448] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.147946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148147] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148283] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C21-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.148316] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C21-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=558, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.148410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.148708] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.149101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.149390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.149551] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.150046] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AD9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.151071] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=9] ====== check clog disk timer task ====== [2025-04-03 14:01:16.151119] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=35] get_disk_usage(ret=0, capacity(MB):=9216, used(MB):=563) [2025-04-03 14:01:16.151316] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=33] cannot_recycle_log_size statistics(cannot_recycle_log_size=322182499, threshold=2899102924, need_update_checkpoint_scn=false) [2025-04-03 14:01:16.151600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.151624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.151713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.151723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.151972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.152124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.152281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.152785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.153165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.153285] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18205][T1002_RFLWorker][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.153318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.153429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.153972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.154125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.154285] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.154449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.154831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.154921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.155086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.155220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.155392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.155575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.155723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.155888] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.156041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.156206] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.156397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.156725] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.156906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.157027] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.157381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.157529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.157707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.158011] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.158202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.158326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.158831] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C22-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.159678] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.169370] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C23-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.169409] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C23-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=444, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.171347] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=879] ====== tenant freeze timer task ====== [2025-04-03 14:01:16.175647] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.175670] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28544 [2025-04-03 14:01:16.175711] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28544 [2025-04-03 14:01:16.175734] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.180360] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C24-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.180422] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C24-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=880, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.180839] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.183250] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4678597088, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:16.183322] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=65] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:16.185464] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.185490] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28546 [2025-04-03 14:01:16.185532] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28546 [2025-04-03 14:01:16.185548] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.191435] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C25-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.191498] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C25-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=923, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.195931] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:16.195974] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28548 [2025-04-03 14:01:16.196058] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28548 [2025-04-03 14:01:16.196078] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.203171] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C26-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.203299] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C26-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1625, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.206658] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.206682] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28550 [2025-04-03 14:01:16.206728] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28550 [2025-04-03 14:01:16.206738] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.208772] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.208839] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=58][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.208859] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.208874] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.214053] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C27-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.214122] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C27-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=633, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.218930] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.225138] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C28-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.225210] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C28-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=887, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.229167] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 513.00; ) [2025-04-03 14:01:16.229227] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:16.229344] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:16.230594] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.230618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.230664] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.230710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.230756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.230867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.230914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.230959] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231468] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.231999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232180] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232653] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.232801] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FB-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.234010] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.234052] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076233845]}) [2025-04-03 14:01:16.234361] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076234065]}) [2025-04-03 14:01:16.234670] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076234374]}) [2025-04-03 14:01:16.235670] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076234681]}) [2025-04-03 14:01:16.236147] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.236162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.236238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.236334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.236467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.236575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.236631] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660076235885]}) [2025-04-03 14:01:16.236901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237203] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C29-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.237252] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C29-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1846, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.237295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237586] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237689] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237827] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.237994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.238719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.239069] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2ADB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.242326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.243140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.259756] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.265413] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE376-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.266938] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.269329] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.269342] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28556 [2025-04-03 14:01:16.269369] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28556 [2025-04-03 14:01:16.269390] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.269495] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C2C-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.269526] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C2C-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=276, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.271866] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:16.272267] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E1-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.275427] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E1-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.279081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E1-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.279151] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.280905] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=49] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660072276946, last_first_register_time_us=1743648235473854, register_time_us=1743660072276946, curr_time_us=1743660076280902) [2025-04-03 14:01:16.281180] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=70][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.281222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.281638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.281859] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.282083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.282762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.283079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.283397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.283407] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.285616] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.285670] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.20.0.5:2882") [2025-04-03 14:01:16.285709] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.286406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.290657] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C2E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.290722] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C2E-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=557, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.293487] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=5) [2025-04-03 14:01:16.302291] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=28] ====== check clog disk timer task ====== [2025-04-03 14:01:16.302319] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=1024, used(MB):=255) [2025-04-03 14:01:16.302345] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=10] cannot_recycle_log_size statistics(cannot_recycle_log_size=200526433, threshold=322122547, need_update_checkpoint_scn=false) [2025-04-03 14:01:16.302473] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C2F-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.302544] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C2F-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1664, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.308848] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.308885] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.308900] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.308912] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.314109] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C30-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.314184] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C30-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1476, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.314396] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:16.314986] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash time detail, (compute_wash_size_time=214, refresh_score_time=110, wash_time=18) [2025-04-03 14:01:16.315072] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=64, clean_start_pos=3145700, clean_num=62914) [2025-04-03 14:01:16.317325] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.317357] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28566 [2025-04-03 14:01:16.317431] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28566 [2025-04-03 14:01:16.317449] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.319089] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.320400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.320806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.321230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.321419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.321578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.321900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.322208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.322373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.322526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.322740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.322985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.323191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.323746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.323951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.324197] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.324450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.324623] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.324873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.325107] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C31-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.325142] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C31-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=802, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.325153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.325388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.325600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.325867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.326243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.326580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.326743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.327010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.327238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.327362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.327518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.327742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.328041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.328166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.328309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.333122] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=52] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:16.334000] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18111][T1002_LogUpdate][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.335505] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C32-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.336447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2ADF-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.338054] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.338097] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076336952]}) [2025-04-03 14:01:16.339922] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076339198]}) [2025-04-03 14:01:16.340737] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076339954]}) [2025-04-03 14:01:16.347063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE0-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.349463] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [18240][T1002_TabletShe][T1002][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.352252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE1-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.354979] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [18779][T1_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=55] ps cache evict(tenant_id=1, stmt_id_map_.size()=1, stmt_info_map_.size()=1, op.get_used_size()=1498, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=1) [2025-04-03 14:01:16.355075] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE1-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.355135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE1-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.355184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE1-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.355228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE1-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.356327] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C34-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.359113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE2-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.359853] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.363687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.363726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.369771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.369839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.372597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.375342] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE5-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.375436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE5-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.375513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE5-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.375853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE5-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.379114] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28572 [2025-04-03 14:01:16.380971] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.381460] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AE5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.383444] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4678797204, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:16.383492] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=41] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:16.385752] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:16.385775] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28574 [2025-04-03 14:01:16.385804] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28574 [2025-04-03 14:01:16.385819] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.389099] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.389113] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28576 [2025-04-03 14:01:16.389139] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28576 [2025-04-03 14:01:16.389144] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.389598] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C37-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.389678] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C37-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1148, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.400933] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C38-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.401006] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C38-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1110, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.402980] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=0] ====== checkpoint timer task ====== [2025-04-03 14:01:16.403294] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.403309] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28578 [2025-04-03 14:01:16.403340] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28578 [2025-04-03 14:01:16.403347] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.405327] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=59] succ to get rec scn(*this={this:0x2b7335a04030, ls_id:{id:1}, tenant_id:1, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:1743616801123802618, v:0}, last_push_gc_task_ts_:1743648235321153, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-04-03 14:01:16.405379] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=51] tx ctx memtable get rec scn(this={ObITable:{this:0x2b73193f0080, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b73193f0080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-04-03 14:01:16.405434] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:791) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=43] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:18446744073709551615, v:3}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:18446744073709551615, v:3}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-04-03 14:01:16.405634] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=29] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2b7398e04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616801173178850, v:0}, clog_base_lsn:{lsn:13286744064}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1743618797082332550, v:0}, all_id_meta:{id_meta:[{limited_id:1743655279117594735, latest_log_ts:{val:1743655262469466896, v:0}}, {limited_id:49000001, latest_log_ts:{val:1743605935907428857, v:0}}, {limited_id:360000001, latest_log_ts:{val:1743640357777174079, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:159, palf_env_:0x2b730a3fc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:7936, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:16.405749] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:523) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=107] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="DATA_CHECKPOINT_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x2b7398e12c50}, min_rec_scn={val:1743616801173178850, v:0}, ls_id_={id:1}) [2025-04-03 14:01:16.405792] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=30] get rec log scn(service_type_=0, rec_log_ts={val:1743655262469466896, v:0}) [2025-04-03 14:01:16.405827] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=34] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-04-03 14:01:16.405842] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:307) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-04-03 14:01:16.405867] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:158) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1743616801173178850, v:0}, checkpoint_scn_in_ls_meta={val:1743616801173178850, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-04-03 14:01:16.405905] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1634) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.405930] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:13286744064}, ls->get_ls_id()={id:1}) [2025-04-03 14:01:16.405946] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:191) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-04-03 14:01:16.408950] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.409004] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.409075] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=64][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.409126] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.412964] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C39-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.413089] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C39-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1873, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.419170] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.425133] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3A-0-0] [lt=1][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.425206] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3A-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.425281] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3A-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2051, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.436669] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.436727] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3B-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1302, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.438601] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE6-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.446783] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076445678]}) [2025-04-03 14:01:16.447454] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3C-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.447490] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3C-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=626, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.447803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE6-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.449152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.450573] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660076449441]}) [2025-04-03 14:01:16.462129] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=39][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.469226] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.469283] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3E-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=649, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.473377] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18504][T1004_TenantInf][T1003][YB42AC140005-000631D6C04D1190-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.479234] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.479253] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28584 [2025-04-03 14:01:16.479304] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28584 [2025-04-03 14:01:16.479316] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.480142] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3F-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.480197] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C3F-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=779, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.481068] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=75][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.489758] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.489829] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28586 [2025-04-03 14:01:16.491866] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.491883] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28588 [2025-04-03 14:01:16.491918] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28588 [2025-04-03 14:01:16.491927] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.493859] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=35] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=4) [2025-04-03 14:01:16.496846] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.496864] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28590 [2025-04-03 14:01:16.496918] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28590 [2025-04-03 14:01:16.496926] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.502152] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C41-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.502233] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C41-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1148, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.508995] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.509180] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=87][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.510565] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.510636] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.510663] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.510687] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.513338] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C42-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.513436] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C42-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1018, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.515726] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:16.516384] INFO [CLOG] operator() (ob_log_replay_service.cpp:1589) [18393][T1003_ReplayPro][T1003][Y0-0000000000000000-0-0] [lt=1190] get_replay_process success(id={id:1}, replayed_log_size=200526433, unreplayed_log_size=0) [2025-04-03 14:01:16.516431] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=60] Wash time detail, (compute_wash_size_time=231, refresh_score_time=132, wash_time=18) [2025-04-03 14:01:16.516488] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:159) [18393][T1003_ReplayPro][T1003][Y0-0000000000000000-0-0] [lt=37] dump tenant replay process(tenant_id=1003, unreplayed_log_size(MB)=0, estimate_time(second)=0, replayed_log_size(MB)=191, last_replayed_log_size(MB)=191, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-04-03 14:01:16.518494] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1][YB42AC140005-000631D6BBFCD1ED-0-0] [lt=27][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.519258] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=61][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.520528] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.520579] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=43] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28592 [2025-04-03 14:01:16.520653] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28592 [2025-04-03 14:01:16.520682] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.521654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1][YB42AC140005-000631D6BBFCD1ED-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.522093] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17414][ServerGTimer][T1][YB42AC140005-000631D6BBFCD1ED-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.522308] INFO [SQL] check_session_leak (ob_sql_session_mgr.cpp:539) [17414][ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=0] get current session count(used_session_count=0, hold_session_count=0, session_leak_count_threshold=100) [2025-04-03 14:01:16.523719] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=7258, clean_start_pos=0, clean_num=62914) [2025-04-03 14:01:16.525050] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C43-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.525151] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C43-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1532, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.529963] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:16.530053] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=81] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28594 [2025-04-03 14:01:16.530760] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28594 [2025-04-03 14:01:16.530861] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=95] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.536347] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C44-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.536421] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C44-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1012, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.547615] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C45-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.547685] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C45-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1052, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.551907] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076551057]}) [2025-04-03 14:01:16.551952] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=42] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076551057]}) [2025-04-03 14:01:16.552397] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:16.552446] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28596 [2025-04-03 14:01:16.552520] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28596 [2025-04-03 14:01:16.552508] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076551982]}) [2025-04-03 14:01:16.552546] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.552998] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076552548]}) [2025-04-03 14:01:16.555219] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076553049]}) [2025-04-03 14:01:16.557975] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660076555633]}) [2025-04-03 14:01:16.558536] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C46-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.558653] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C46-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=774, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.562268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE9-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.562412] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.565397] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4811479367, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:16.568145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AE9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.576663] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.580639] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.580608] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C48-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.580684] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=36] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28600 [2025-04-03 14:01:16.580693] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C48-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=895, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.580757] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28600 [2025-04-03 14:01:16.580774] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.580962] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=0] ====== traversal_flush timer task ====== [2025-04-03 14:01:16.581145] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=61][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.581599] INFO [MDS] flush (mds_table_mgr.cpp:172) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=26] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b739dad0510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616943687716218, v:0}, clog_base_lsn:{lsn:4227600384}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743618994719735223, v:0}, all_id_meta:{id_meta:[{limited_id:1743655491736477739, latest_log_ts:{val:1743655475118797847, v:0}}, {limited_id:22000001, latest_log_ts:{val:1743616321186417451, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:624, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:16.581842] INFO [MDS] flush (mds_table_mgr.cpp:195) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=128] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:200001}, rec_scn:{val:1743616894738777400, v:0}}, {tablet_id:{id:200002}, rec_scn:{val:1743616894738777401, v:0}}, {tablet_id:{id:200003}, rec_scn:{val:1743616894738777402, v:0}}, }, max_consequent_callbacked_scn={val:1743655277355487643, v:0}, *this={this:0x2b739db96510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1743616894738777400, v:0}, clog_base_lsn:{lsn:3489447936}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616894738777399, v:0}, all_id_meta:{id_meta:[{limited_id:1740557836670287627, latest_log_ts:{val:1740557820032434136, v:0}}, {limited_id:19000001, latest_log_ts:{val:1740413613052489362, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743590092673837460, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:637, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:16.582224] INFO [MDS] flush (mds_table_mgr.cpp:195) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=285] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:200009}, rec_scn:{val:1743616823359480140, v:0}}, {tablet_id:{id:200006}, rec_scn:{val:1743616823359480141, v:0}}, {tablet_id:{id:200007}, rec_scn:{val:1743616823359480142, v:0}}, }, max_consequent_callbacked_scn={val:1743655265068231480, v:0}, *this={this:0x2b73ab388510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1743616823359480140, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616823359480139, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743590092742141696, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:1, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:16.582386] INFO [MDS] flush (mds_table_mgr.cpp:195) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=89] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:5, ordered_tablets:{{tablet_id:{id:1152921504606846977}, rec_scn:{val:1743616894738777400, v:0}}, {tablet_id:{id:200008}, rec_scn:{val:1743616894738777401, v:0}}, {tablet_id:{id:200005}, rec_scn:{val:1743616894738777402, v:0}}, {tablet_id:{id:1152921504606846978}, rec_scn:{val:1743616894840797864, v:0}}, {tablet_id:{id:200004}, rec_scn:{val:1743616894840797865, v:0}}, }, max_consequent_callbacked_scn={val:1743655479483236415, v:0}, *this={this:0x2b7375de2510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1003}, ls_create_status:1, clog_checkpoint_scn:{val:1743616894738777400, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616894738777399, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743614587358473541, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:14, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:5505024, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:16.582491] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=101] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-04-03 14:01:16.583662] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4678997438, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:16.583729] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=59] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:16.589841] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:16.589898] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=49] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28602 [2025-04-03 14:01:16.590005] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28602 [2025-04-03 14:01:16.590061] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=51] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.592759] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C49-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.592844] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C49-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1952, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.593052] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17690][T1_Occam][T1][YB42AC140005-000631D6BCCCDA21-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.594080] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17690][T1_Occam][T1][YB42AC140005-000631D6BCCCDA21-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.603256] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.603324] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=58] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28604 [2025-04-03 14:01:16.603405] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28604 [2025-04-03 14:01:16.603428] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.604527] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4A-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.604598] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4A-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1544, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.606158] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.609083] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.609138] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.609158] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.610533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.610742] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.616489] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4B-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.616553] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4B-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1786, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.619302] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=64][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.627357] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4C-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.627426] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4C-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=709, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.632668] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.632725] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=49] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28606 [2025-04-03 14:01:16.632789] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28606 [2025-04-03 14:01:16.632853] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=60] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.637987] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4D-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.638058] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4D-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=493, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.644480] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=202] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:16.648983] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.649090] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4E-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=887, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.651095] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.651460] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.651485] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28608 [2025-04-03 14:01:16.651561] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28608 [2025-04-03 14:01:16.651569] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.659497] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076658699]}) [2025-04-03 14:01:16.659914] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4F-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.659988] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C4F-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=722, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.660078] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076659552]}) [2025-04-03 14:01:16.660110] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=30] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076659552]}) [2025-04-03 14:01:16.660849] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076660152]}) [2025-04-03 14:01:16.662245] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4811576902, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:16.662297] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:16.663937] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076660900]}) [2025-04-03 14:01:16.667450] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660076664411]}) [2025-04-03 14:01:16.670607] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.674106] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28614 [2025-04-03 14:01:16.683886] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] do_fetch_log_ finished(ret=0, palf_id=1, self="172.20.0.5:2882", dest="172.20.0.4:2882", fetch_log_count=2048, fetch_start_lsn={lsn:13483027323}, prev_lsn={lsn:13483026786}, fetch_start_log_id=30010186, last_slide_log_id=30010185, fetch_log_size=41943040, accepted_mode_pid=159, last_fetch_req_time=1743660076683884, last_fetch_end_lsn={lsn:13524970363}, last_fetch_max_log_id=30012233, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:30010310, max_lsn:13483085678, max_scn:{val:1743655277903270387, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:13483027323}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:13441142638}}, last_submit_lsn:{lsn:13483026786}, last_submit_end_lsn:{lsn:13483027323}, last_submit_log_id:30010185, last_submit_log_pid:159, max_flushed_lsn:{lsn:13483026786}, max_flushed_end_lsn:{lsn:13483027323}, max_flushed_log_pid:159, committed_end_lsn:{lsn:13483027323}, last_slide_log_id:30010185, last_slide_scn:{val:1743655265084940962, v:0}, last_slide_lsn:{lsn:13483026786}, last_slide_end_lsn:{lsn:13483027323}, last_slide_log_pid:159, last_slide_log_accum_checksum:1426087181, last_fetch_end_lsn:{lsn:13524970363}, last_fetch_max_log_id:30012233, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660076683884, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7398e68530}) [2025-04-03 14:01:16.684006] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=110][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.685455] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.688739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AEC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.691240] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.691268] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28620 [2025-04-03 14:01:16.691315] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28620 [2025-04-03 14:01:16.691329] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.693930] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=3) [2025-04-03 14:01:16.694548] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C52-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.694631] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C52-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1094, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.695996] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=43] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:16.696030] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28622 [2025-04-03 14:01:16.696082] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28622 [2025-04-03 14:01:16.696095] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.705767] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=27] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:16.705844] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C53-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.709193] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.709251] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=51][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.709282] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.709329] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.717109] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C54-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.717181] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C54-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1104, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.717779] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.717801] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28624 [2025-04-03 14:01:16.717848] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28624 [2025-04-03 14:01:16.717862] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.719404] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.721945] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.721964] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28626 [2025-04-03 14:01:16.722128] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28626 [2025-04-03 14:01:16.722141] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.724315] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:16.724861] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash time detail, (compute_wash_size_time=211, refresh_score_time=100, wash_time=17) [2025-04-03 14:01:16.728529] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C55-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.728605] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C55-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1201, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.731491] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=6582, clean_start_pos=62914, clean_num=62914) [2025-04-03 14:01:16.732113] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:520) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] [CACHE] tenant_id= 1 cache memory info: [CACHE] tenant_id= 1 | cache_name= schema_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 194 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= vtable_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 7 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 29 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 79 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 208 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 73 | hold_size= 0 [CACHE] tenant_id= 1 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 557 | hold_size= 0 [2025-04-03 14:01:16.732250] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:520) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=48] [CACHE] tenant_id= 1001 cache memory info: [CACHE] tenant_id= 1001 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 75 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 47 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 124 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 55 | hold_size= 0 [CACHE] tenant_id= 1001 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 335 | hold_size= 0 [2025-04-03 14:01:16.732352] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:520) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] [CACHE] tenant_id= 1002 cache memory info: [CACHE] tenant_id= 1002 | cache_name= index_block_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_map_size= 2079744 | kv_cnt= 202 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= user_block_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_map_size= 2079744 | kv_cnt= 5 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 96 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 49 | hold_size= 0 [CACHE] tenant_id= 1002 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 260 | hold_size= 0 [2025-04-03 14:01:16.732453] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:520) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] [CACHE] tenant_id= 1003 cache memory info: [CACHE] tenant_id= 1003 | cache_name= index_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 15 | hold_size= 0 [CACHE] tenant_id= 1003 | cache_name= user_block_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 59 | hold_size= 0 [CACHE] tenant_id= 1003 | cache_name= storage_meta_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_map_size= 2079744 | kv_cnt= 115 | hold_size= 0 [CACHE] tenant_id= 1003 | cache_name= tx_data_kv_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 2 | hold_size= 0 [CACHE] tenant_id= 1003 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 60 | hold_size= 0 [CACHE] tenant_id= 1003 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 379 | hold_size= 0 [2025-04-03 14:01:16.732564] INFO print_tenant_cache_info (ob_kvcache_inst_map.cpp:520) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] [CACHE] tenant_id= 1004 cache memory info: [CACHE] tenant_id= 1004 | cache_name= index_block_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_map_size= 2079744 | kv_cnt= 584 | hold_size= 0 [CACHE] tenant_id= 1004 | cache_name= user_block_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_map_size= 2079744 | kv_cnt= 18 | hold_size= 0 [CACHE] tenant_id= 1004 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 728 | hold_size= 0 [CACHE] tenant_id= 1004 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 46 | hold_size= 0 [CACHE] tenant_id= 1004 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 267 | hold_size= 0 [2025-04-03 14:01:16.732602] INFO print_current_status (ob_kvcache_hazard_version.cpp:431) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] [KVCACHE-HAZARD] hazard version status info: current version: 45313 | min_version= 45313 | total thread store count: 313 | total nodes count: 0 | [CACHE] tenant_id= 1004 | cache_name= index_block_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_map_size= 2079744 | kv_cnt= 584 | hold_size= 0 [CACHE] tenant_id= 1004 | cache_name= user_block_cache | cache_size= 4160512 | cache_store_size= 2080768 | cache_map_size= 2079744 | kv_cnt= 18 | hold_size= 0 [CACHE] tenant_id= 1004 | cache_name= storage_meta_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 728 | hold_size= 0 [CACHE] tenant_id= 1004 | cache_name= opt_table_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 46 | hold_size= 0 [CACHE] tenant_id= 1004 | cache_name= opt_column_stat_cache | cache_size= 6241280 | cache_store_size= 4161536 | cache_map_size= 2079744 | kv_cnt= 267 | hold_size= 0 [2025-04-03 14:01:16.736167] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=40] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:16.740126] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C56-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.740218] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C56-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1283, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.747609] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:16.747638] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28628 [2025-04-03 14:01:16.747688] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28628 [2025-04-03 14:01:16.747699] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:16.748932] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=39] start do ls ha handler(ls_id_array_=[{id:1003}, {id:1}, {id:1001}, {id:1002}]) [2025-04-03 14:01:16.751053] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C57-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.751134] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C57-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=727, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.762449] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.762979] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=36][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.764744] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C58-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.764866] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C58-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2792, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.765495] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.765536] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.765542] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:15810, tid:17828}]) [2025-04-03 14:01:16.765677] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.765699] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.765865] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.765886] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.766353] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.766411] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.766737] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.767366] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4811681553, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:16.767408] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:16.770027] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076768228]}) [2025-04-03 14:01:16.770114] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6BC-0-0] [lt=1016][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.770767] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076770074]}) [2025-04-03 14:01:16.771614] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076770834]}) [2025-04-03 14:01:16.774631] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076771652]}) [2025-04-03 14:01:16.775588] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.775621] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=32][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.775745] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C59-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.775803] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C59-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=738, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.775923] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.775964] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=40][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.776381] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.776405] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.776646] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.776670] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=24][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.776709] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:16.776756] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28630 [2025-04-03 14:01:16.776871] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28630 [2025-04-03 14:01:16.776896] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:16.776943] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.776956] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.776993] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660076775058]}) [2025-04-03 14:01:16.777920] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.777945] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=25][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.778116] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.778129] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.778261] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.778281] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.778432] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.778441] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.778699] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.778722] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=22][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.778961] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.778980] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.779116] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.779128] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.779421] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.779467] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=31][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.779710] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.779753] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=43][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.779839] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.779850] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.779932] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.779944] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780097] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780114] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780196] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780211] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780342] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780356] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780437] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780450] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780527] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780540] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780637] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780645] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780731] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780737] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780865] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780884] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.780959] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.780972] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.781112] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.781127] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.781206] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.781229] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=22][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.781318] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.781330] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.781343] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.781408] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.781414] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.781542] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.781555] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.781678] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.781690] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.781767] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.781779] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.781942] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.781956] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782043] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782051] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782114] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782132] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782214] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782227] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782303] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782315] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.782476] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=19][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782483] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782547] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782560] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782638] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782656] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782732] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782738] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782827] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782839] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.782915] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.782935] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783032] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.783045] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783126] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.783139] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783223] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.783235] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783311] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.783324] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783399] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.783419] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=20][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783576] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.783589] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783754] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.783767] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783773] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4679197525, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:16.783836] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:16.783900] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.783907] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.783995] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.784001] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.784120] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.784129] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.784306] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.784321] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.784493] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.784507] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.784628] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.784641] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.784722] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.784753] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=30][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.784894] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.784908] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.784985] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:16.785316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AED-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.786682] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C5A-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.786747] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C5A-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=713, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.796577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AEF-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.801009] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=56] PNIO [ratelimit] time: 1743660076801006, bytes: 0, bw: 0.000000 MB/s, add_ts: 1006786, add_bytes: 0 [2025-04-03 14:01:16.809280] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.809337] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.809378] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.809396] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.812303] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18235][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.817420] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.819494] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=86][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.820541] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C5D-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.820638] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C5D-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1396, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.832111] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C5E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.832220] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C5E-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1338, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.843316] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C5F-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.843404] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C5F-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=929, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.845758] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:16.845831] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=41] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28634 [2025-04-03 14:01:16.845926] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28634 [2025-04-03 14:01:16.845952] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:16.847316] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [17564][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=33] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.850931] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:16.851114] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.851458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.851638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.851746] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=34] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:16.852418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.852779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.852907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.853612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.853774] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.853938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.854217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.854529] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C60-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.854835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.855371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.855568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.855889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.856085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.856271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.856467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.856740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.857294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.857442] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.857642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.857969] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.858511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.858634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.858844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.859131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.859624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.859786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.859926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.860097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.860395] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.860588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.860789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.860926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.861055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.861562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.862413] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=4811777329, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:16.862942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.863172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.863333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.863792] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.864131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.864286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.864671] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.864771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.864942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.865301] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.865460] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.865585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.865948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.866108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.866243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.866623] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.866906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.867165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.867875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.867894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.868029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.868469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.868639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.868860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.869043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.869353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.869500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.869897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.870025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.870335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.870522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.870641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.871295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.871539] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.871906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.872000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.872100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.872260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.872491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.872585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.872750] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.872931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.873108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.873408] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.873582] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.873944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF1-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875336] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.875868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.876157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.876167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.876353] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C62-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.878143] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076877737]}) [2025-04-03 14:01:16.878504] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076878168]}) [2025-04-03 14:01:16.879560] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076878522]}) [2025-04-03 14:01:16.879799] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.879849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=49] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.880117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.880139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.880347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.880372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.880480] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076879581]}) [2025-04-03 14:01:16.880739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.880766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.881106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.881133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.881370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.881391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.881414] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.881643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.881838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.882588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.882614] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.882986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.883024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.883355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.883378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.883890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.883901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.884007] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.884296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.884322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.884482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.884712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.884870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.884970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.885135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.885484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.885639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.886382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.886407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.886581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.886601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.887078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.887104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.887491] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C63-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.887575] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C63-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1008, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.887831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.887854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.888309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.888331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.890119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.890143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.890905] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.890923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.891091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.891107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.891168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.891180] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.891243] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:16.891275] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28636 [2025-04-03 14:01:16.891279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.891338] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28636 [2025-04-03 14:01:16.891740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.892134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.892490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.892647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.892830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.892940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.893116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.893273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.893439] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.893637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.893786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.893944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.898628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.898648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.898724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.898735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.898795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.898804] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.898984] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.898995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899468] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899752] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899761] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.900002] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.899936] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C64-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.900053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.900058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.900063] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C64-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1083, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.900131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.900140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF2-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.900946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.900968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901667] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.901956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.902090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.902249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.902594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.902758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.902889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.903110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.903391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.903544] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.903719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.903876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.904122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.904217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.904537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.904835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.904993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.905245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.905383] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.905489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.905679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.905995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.906160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.906295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.906483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.906649] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.906791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.907133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.907354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.907459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.907770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.907938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.908093] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.908321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.908414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.908586] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.908739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.908955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.909107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.909253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.909385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.909380] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.909416] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.909466] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.909490] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.909596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.909972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.909986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.910316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.910415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.910588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF3-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.911083] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C65-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.911157] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C65-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=887, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.911748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.911762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.911860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.912212] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.912319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.912895] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.912989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.913318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.913475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.913975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.914113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.914294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.914620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.914985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.915116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.915278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.915427] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.915611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.915929] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.916069] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.916225] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.916413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.916574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.916755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.916919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF4-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.917732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.917756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.917864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.918064] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.918217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.918381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.918510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.918740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.918858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.919029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.919143] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=56][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:16.919183] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=38][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660105653097, now=[mts=1743660075723643], now0=[mts=1743660075723643], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:16.919212] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.919213] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=26][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a431d2a0, tx_id:{txid:76175012}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660075723054, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:16.919339] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.919296] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=75][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b739ba5c1a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a431d2a0}) [2025-04-03 14:01:16.919370] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=71][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a431d2a0, tx_id:{txid:76175012}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660075723054, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:0, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:16.919537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.919443] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=70] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.919578] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:16.919576] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=130][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:16.919598] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=19][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:16.919637] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=37][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:16.919649] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=10][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:16.919661] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.919660] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=9][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:16.919678] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=16][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:16.919728] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:16.919795] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=49][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:16.919845] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=49][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:16.919854] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=9][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:16.919949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.919971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.920133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.920314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.920479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.920641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.921007] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.921114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.921274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.921467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.921611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.921783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.921946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.922103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.922250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.922422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.922585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.922948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.923081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF5-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.923292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.923441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.923677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.923759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.923923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.924048] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.924255] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.924531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.924862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.925045] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.925183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.925366] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.925518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.925671] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.925840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.926009] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.926179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.926332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.926496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.926818] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.926976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.927445] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17463][pnio1][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.927460] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.927485] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.927513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:16.927795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.928291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.928445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.928764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF6-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.928927] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.929501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.929600] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28640 [2025-04-03 14:01:16.929758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.929729] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [17917][T1001_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=44] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.930123] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:16.930215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.930406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.930596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.931077] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.931238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.931398] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.932036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.932192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.932346] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.932703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.932907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.932997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.933146] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.933194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.933494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.933651] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.933850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.933987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.934166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.934332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.934467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.934635] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.934826] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.935006] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.935138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF7-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.935934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.935956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.936003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.936018] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.936106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.936296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.936622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.936848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.936941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.937079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.937256] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.937394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.937563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.937790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.937892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.938047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.938218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.938387] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.938538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.938768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.938867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.939028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.939234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.939364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.939558] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.939677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.939842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.940022] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.940336] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.940347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.940675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.940820] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.941018] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.941155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AF8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.943946] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:16.943969] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28642 [2025-04-03 14:01:16.944006] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28642 [2025-04-03 14:01:16.944020] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:16.945386] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C68-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.945462] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C68-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=496, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.949783] WARN alloc_chunk (achunk_mgr.cpp:326) [18700][BlackListServic][T1][YB42AC140005-000631D6C19CC4BE-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit(msg="oops, over total memory limit", hold=10735362048, limit=10737418240) [2025-04-03 14:01:16.956465] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C69-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:16.956531] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C69-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.956595] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C69-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=963, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.962502] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.962516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.962532] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.962542] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.962707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.962718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.962724] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.962728] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.962900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.962917] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.962923] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.962927] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.963078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963095] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.963099] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.963334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963353] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.963357] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.963580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963598] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.963603] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.963736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963759] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.963763] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.963902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.963917] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.963920] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.964052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964066] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.964070] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.964232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964255] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.964259] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.964495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964508] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964514] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.964518] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.964650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964661] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964667] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.964671] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.964736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964749] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.964752] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.964773] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.964859] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964872] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.964876] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.964958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.964972] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.964976] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.965111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.965123] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.965128] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:16.965132] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:16.965221] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4811879775, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:16.965237] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:16.966145] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=63][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4621, tid:17533}]) [2025-04-03 14:01:16.967726] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.967771] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=43][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:16.967786] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=14][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:16.967855] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=67][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:16.967875] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=12][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:16.967944] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=67] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.967999] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=54][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.968085] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=84] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1279, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.968128] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6A-0-0] [lt=41][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.969668] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1002}]) [2025-04-03 14:01:16.979069] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=62][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.979128] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=58][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:16.979144] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=15][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:16.979160] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=14][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:16.979198] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=30][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:16.979253] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=53] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.979289] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=35][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.979339] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=48] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=999, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.979376] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6B-0-0] [lt=35][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.981496] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:16.982953] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.983002] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:16.983036] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660076981507], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.983065] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660076981507], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.983119] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660076981507]}) [2025-04-03 14:01:16.983871] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.983894] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:16.983908] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660076983148], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.983927] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660076983148], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.983961] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4679397540, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:16.983985] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660076983148]}) [2025-04-03 14:01:16.983996] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:16.984687] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.984721] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:16.984735] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660076984034], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.984754] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660076984034], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.984789] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660076984034]}) [2025-04-03 14:01:16.985653] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.985674] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:16.985687] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660076984842], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.985704] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660076984842], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.985727] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660076984842]}) [2025-04-03 14:01:16.986822] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:16.986853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.986867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.987068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.987092] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.987084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.987115] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:16.987151] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660076986185], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.987171] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660076986185], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:16.987198] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660076986185]}) [2025-04-03 14:01:16.987346] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.987355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.987624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.987634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.987879] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.987887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.988170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.988190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.988499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.988518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.988656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.988689] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.989055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.989071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.989360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.989370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.989640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.989649] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.989847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.989855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990427] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=39][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.990465] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=36][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:16.990478] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=11][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:16.990489] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=9][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:16.990530] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=33][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:16.990587] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=55] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.990648] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=59][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.990680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990686] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990692] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=42] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1053, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:16.990739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990729] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6C-0-0] [lt=35][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:16.990911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.990924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991195] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991356] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.991941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992778] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.992945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993072] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993094] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:16.993103] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8][errcode=-4013] rpc memory pool alloc memory failed(sz=1340, alloc_sz=16256) [2025-04-03 14:01:16.993109] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=5][errcode=-4013] alloc buffer fail(payload_sz=1156) [2025-04-03 14:01:16.993115] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=5][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:16.993139] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=18][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=994834, timeout_timestamp=1743660077984876) [2025-04-03 14:01:16.993155] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=14][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:16.993162] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=5][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:16.993168] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:16.993175] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=5][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:16.993180] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=4][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:16.993185] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=4][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:16.993194] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:16.993203] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=7][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:16.993238] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=25][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:16.993243] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=5][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:16.993263] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=20][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:16.993359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.993844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.994076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.994127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=49] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.994327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.994351] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.994483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.994506] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.994801] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:16.994839] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28644 [2025-04-03 14:01:16.995489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.995540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=49] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.995747] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28644 [2025-04-03 14:01:16.995763] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:16.995796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.995837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.996122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.996151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.996471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.996496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.996619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.996642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.996784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.996806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997120] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.997910] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998077] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998833] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.998855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.999003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.999040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.999246] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.999268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.999390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.999410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.999677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:16.999699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.000084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.000106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.007038] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1962) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=5] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-04-03 14:01:17.007066] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1464) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=23] start lib cache evict(tenant_id=1002, mem_hold=9027584, mem_limit=26843545, cache_obj_num=34, cache_node_num=34) [2025-04-03 14:01:17.007079] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1481) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=9027584, mem_limit=26843545, cache_obj_num=34, cache_node_num=34) [2025-04-03 14:01:17.007088] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2694) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=8] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:17.007781] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=40][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.007830] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=47][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:17.007844] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=13][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:17.007854] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=9][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.007867] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=8][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:17.007908] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=40] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.007954] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=45][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.007980] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=25] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=670, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.008040] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6D-0-0] [lt=58][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.009553] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [17844][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=874] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:17.009593] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.009609] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.009618] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.009626] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.010697] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2449) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=34, alloc_obj_list=[{obj_id:1, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175493}, {obj_id:2, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176543}, {obj_id:10, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:175828}, {obj_id:11, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:180892}, {obj_id:23, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:181961}, {obj_id:43, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187646}, {obj_id:44, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185047}, {obj_id:45, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:188001}, {obj_id:46, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185069}, {obj_id:47, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:187994}, {obj_id:48, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:185011}, {obj_id:49, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191261}, {obj_id:50, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:186550}, {obj_id:51, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182609}, {obj_id:52, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:209465}, {obj_id:56, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:191136}, {obj_id:57, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:182721}, {obj_id:58, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183848}, {obj_id:60, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:253856}, {obj_id:64, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176578}, {obj_id:65, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:173048}, {obj_id:66, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:174057}, {obj_id:67, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:183126}, {obj_id:68, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:178338}, {obj_id:69, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:244258}, {obj_id:70, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:224434}, {obj_id:71, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:220628}, {obj_id:72, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:274876}, {obj_id:73, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:214173}, {obj_id:74, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:212788}, {obj_id:75, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:205532}, {obj_id:76, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:196291}, {obj_id:77, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:190887}, {obj_id:78, tenant_id:1002, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:2, added_to_lc:true, mem_used:176654}]) [2025-04-03 14:01:17.010815] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2702) [18131][T1002_PlanCache][T1002][Y0-0000000000000000-0-0] [lt=107] schedule next cache evict task(evict_interval=5000000) [2025-04-03 14:01:17.012912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.012947] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013147] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.013926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014057] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.014927] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015383] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.015937] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016064] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.016994] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:17.017027] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=0][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:17.017048] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=18][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:17.017057] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=6][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.017066] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=8][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:17.017072] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=6][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:17.017079] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=6][errcode=-4013] query failed(ret=-4013, conn=0x2b73a2330050, start=1743660076984886, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:17.017093] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=11][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.017183] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=9][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:17.017198] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=13][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1002, ls_ids=[{id:1}]) [2025-04-03 14:01:17.017213] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=13][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1002, ls_ids=[{id:1}]) [2025-04-03 14:01:17.017232] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=17][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1002, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660076984769}) [2025-04-03 14:01:17.017250] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=17][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:17.017258] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AF9-0-0] [lt=7][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:17.018449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.018469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.018545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.018684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.018995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.019322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.019643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.019801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.019690] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.019964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.020170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.020659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.020786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.020967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.021085] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.021096] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=10][errcode=-4013] rpc memory pool alloc memory failed(sz=1374, alloc_sz=16256) [2025-04-03 14:01:17.021103] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1190) [2025-04-03 14:01:17.021108] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.021117] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1003, svr="172.20.0.3:2882", timeout=997793, timeout_timestamp=1743660078017306) [2025-04-03 14:01:17.021126] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=8][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:17.021133] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=5][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:17.021139] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:17.021144] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:17.021155] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=9][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:17.021161] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=5][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:17.021168] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=6][errcode=-4013] execute failed(ret=-4013, tenant_id=1003, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:17.021179] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=9][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:17.021201] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=14][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.021206] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.021209] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=3][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:17.021304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.021313] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.021375] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6E-0-0] [lt=26][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.021402] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6E-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:17.021454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.021454] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6E-0-0] [lt=51][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:17.021467] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6E-0-0] [lt=12][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.021485] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6E-0-0] [lt=9][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:17.021532] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6E-0-0] [lt=1][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.021602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.021590] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6E-0-0] [lt=1][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.021764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.022270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.022441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.022581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.022766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.022912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.023243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.023402] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.023550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.023712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.023874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.024141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.024237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.024392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.024529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.024740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.024869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.025109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.025131] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:17.025139] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=6][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:17.025147] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=6][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:17.025152] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.025164] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=11][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1003) [2025-04-03 14:01:17.025168] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1003) [2025-04-03 14:01:17.025173] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] query failed(ret=-4013, conn=0x2b739163a050, start=1743660077017315, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:17.025181] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=6][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.025235] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=6][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004) [2025-04-03 14:01:17.025246] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=9][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:17.025257] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=10][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:17.025265] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=7][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1004, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660076987187}) [2025-04-03 14:01:17.025277] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=11][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:17.025282] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2AFA-0-0] [lt=4][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:17.032419] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=49][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.032472] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=51][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:17.032487] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=14][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:17.032498] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=10][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.032513] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=9][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:17.032556] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=42] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.032586] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=29][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.032634] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=46] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=861, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.032679] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C6F-0-0] [lt=44][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.033593] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033741] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033818] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.033935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034065] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034126] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034416] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034725] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034782] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034910] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.034964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.035312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.035765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.035964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.036131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.036438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.036778] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.036899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.037233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.037415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.037584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.037749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.038038] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.038232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.038386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.038536] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.038715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.039038] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.039214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.039370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.043572] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=48][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.043607] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=34][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:17.043617] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=9][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:17.043626] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=8][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.043639] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=8][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:17.043674] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=34] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.043700] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=25][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.043744] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=42] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=823, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.043768] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C70-0-0] [lt=23][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.052063] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.052116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=48] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.052134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.052284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.052311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.052564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.052763] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=197] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.053279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.053323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.053780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.053806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.054124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.054140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.054485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.054503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.054555] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C71-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.054617] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C71-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=686, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.055183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.055201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.055685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.055704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.055965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.055982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.056208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.056223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.056493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.056509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.056728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.056743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.057280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.057300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.057607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.057625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.057834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.057850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.058117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.058137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.058379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.058398] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.059975] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:103) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=36] tx gc loop thread is running(MTL_ID()=1004) [2025-04-03 14:01:17.060443] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=84] update calc upper info once(calc_upper_info_={min_start_scn_in_ctx_:{val:1743655471770425554, v:0}, keep_alive_scn_:{val:1743655471770425554, v:0}, update_ts_:1743660077060440}) [2025-04-03 14:01:17.060488] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=32] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-04-03 14:01:17.060529] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=29] get_min_unreplayed_log_info(lsn={lsn:4316157182}, scn={val:1743655475423544391, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1743660076783959, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:4316157182}, next_to_submit_scn_:{val:1743655475423544391, v:0}, base_lsn_:{lsn:4227600384}, base_scn_:{val:1743616803524366216, v:0}, iterator_:{iterator_impl:{buf_:0x2b738be8d000, next_round_pread_size:2121728, curr_read_pos:38458, curr_read_buf_start_pos:0, curr_read_buf_end_pos:38458, log_storage_:{IteratorStorage:{start_lsn:{lsn:4316118724}, end_lsn:{lsn:4316157182}, read_buf:{buf_len_:487424, buf_:0x2b738be8d000}, block_size:67104768, log_storage_:0x2b736e5c1bf0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1743655475423544390, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:34, scn_:{val:1743655475423544390, v:0}, data_checksum:2923900223, flag:1}}, init_mode_version:0, accumulate_checksum:1508656143, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1743655475423544390, v:0}}}}}) [2025-04-03 14:01:17.061937] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.062400] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=108] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616943687716218, v:0}, clog_base_lsn:{lsn:4227600384}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743618994719735223, v:0}, all_id_meta:{id_meta:[{limited_id:1743655491736477739, latest_log_ts:{val:1743655475118797847, v:0}}, {limited_id:22000001, latest_log_ts:{val:1743616321186417451, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:624, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:17.062613] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=4811977595, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.062644] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.062677] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=96] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.063718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.063739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.063982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.064194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.064205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.064628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.064867] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=61][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.064932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.065076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.065357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.065690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.065837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.066163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.066397] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.066413] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=44][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:8078, tid:18713}]) [2025-04-03 14:01:17.066584] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.066620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.066830] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.066847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.066911] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.067187] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.067209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.067413] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.067426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.067715] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=0] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1004, *ls_ptr={ls_meta:{tenant_id:1004, ls_id:{id:1003}, ls_create_status:1, clog_checkpoint_scn:{val:1743616894738777400, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616894738777399, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743614587358473541, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:14, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:5505024, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:17.067967] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.068201] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.068232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.068438] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.069093] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.069120] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.069129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.069332] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.069372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.069637] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.069660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.069914] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.069951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.070062] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.070173] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.070373] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.070395] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=21] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.070534] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.070698] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.070855] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.070895] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.071206] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.071222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.071382] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.071412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.071584] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.071761] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.071949] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.071963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.072145] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.072173] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.072294] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.072522] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.072553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.072682] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.072709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.072916] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.073092] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.073122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.073286] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.073300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.073418] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.073582] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.073595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.073715] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.073852] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.073864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.074090] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.074237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=144] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.074559] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.074574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.074707] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.074739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.074944] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.075091] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.075215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=122] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.075399] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.075413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.075579] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.075706] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.075718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.075858] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.075969] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.076036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=65] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.076158] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.076352] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.076379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.076524] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.076536] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.076639] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.076964] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.076920] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C73-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.077150] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.077177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.077300] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.077458] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.077779] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.077909] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.077934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.078062] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.078169] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.078357] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.078371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.078481] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.078599] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.078611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.078725] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.078856] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.078868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.079178] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.079206] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.079375] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.079516] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.079630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=112] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.079841] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.080123] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.080150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.080316] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.080877] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.080999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=120] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.081044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=44] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.081222] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.081238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D8-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.081563] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.084080] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.084137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=53] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084242] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.084261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084419] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.084454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084565] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.084593] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084674] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.084701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084804] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.084867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=61] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.084974] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.084999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.085123] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.085480] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.085560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=62] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.085691] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.085913] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.085943] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.086069] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.086154] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.086215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=59] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.086304] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.086392] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.086436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=42] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.086495] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.086582] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.086692] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.086717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.088353] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C74-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.088419] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C74-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1284, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.089246] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077087932]}) [2025-04-03 14:01:17.091215] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660077089285]}) [2025-04-03 14:01:17.092132] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660077091254]}) [2025-04-03 14:01:17.093035] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660077092153]}) [2025-04-03 14:01:17.094134] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=2] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=1) [2025-04-03 14:01:17.094348] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660077093332]}) [2025-04-03 14:01:17.094608] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.094675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=63] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.094695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.094850] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.094892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.094905] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.095081] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.095113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.095126] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.095245] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.095288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.095301] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.095567] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.095611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.095624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.095762] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.095789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.095801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.096047] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.096086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.096099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.096248] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.096281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.096291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.096520] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.096588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=65] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.096603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.096737] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.096997] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.097047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=46] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.097072] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.097269] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.097716] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.097762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=43] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.097776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.097986] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.098034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=44] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.098279] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.098496] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.098517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.098710] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.098864] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.098894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.099045] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.099084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=37] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.099223] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.099387] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.099425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.099689] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.099705] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C75-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.099895] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.099926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.100108] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.100150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.100286] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.100401] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.100657] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.100803] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.100862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=56] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.101005] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.101295] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.101522] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.101549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.101701] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.101921] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.101936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.102008] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.102112] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.102187] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.102378] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.102400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.102522] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.102703] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.102792] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.102829] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=36] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.102909] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103042] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.103180] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103282] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103387] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103409] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.103605] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.103793] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103877] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=21] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.103909] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.103929] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.104094] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.104162] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.104229] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.104261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.104368] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.104450] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.104453] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.104569] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:17.104795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.105276] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.105577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.105735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.105945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.106262] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.106543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.106901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.107095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.107406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.107587] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.107715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.107856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.108036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.108500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.108655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.108921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.109662] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660073105762, last_first_register_time_us=1743648246712872, register_time_us=1743660073105762, curr_time_us=1743660077109661) [2025-04-03 14:01:17.109731] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.109845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.109977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.110150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.110287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.110464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.110615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.110667] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.110689] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.110698] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.110708] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.110772] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.110986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.111303] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.111421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.111608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.111785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.111892] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.111919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.112145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.112245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.112581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.112729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.113098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.113248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.113559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.113716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.113888] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.114039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.114377] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.114679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.114865] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.115051] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.115172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.115509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.116173] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.116339] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.116476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.117160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2AFD-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.117461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.117609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.117787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.118112] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.118352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.118835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.119156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.119547] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.119743] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=162][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.119960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.120413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.120916] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C77-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.122751] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.122982] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=221] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56320 [2025-04-03 14:01:17.123112] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56320 [2025-04-03 14:01:17.123137] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.131577] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C78-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.131626] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C78-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=536, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.137591] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=1] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:17.138095] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=50] Wash time detail, (compute_wash_size_time=196, refresh_score_time=97, wash_time=15) [2025-04-03 14:01:17.142858] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C79-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.142928] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C79-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=791, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.143591] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=119] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:17.144516] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=6363, clean_start_pos=188742, clean_num=62914) [2025-04-03 14:01:17.153762] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7A-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.153851] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7A-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=720, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.162958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.163000] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:17.163052] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.163752] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.163782] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:17.163797] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.164100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.164123] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:17.164151] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.164634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.164660] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:17.164684] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.164951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.164973] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1, wash_size=6291456) [2025-04-03 14:01:17.164951] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.165198] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.165280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.165277] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7B-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1105, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.165515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.166041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.166459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.167186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.167510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.167666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.167876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.168125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.168278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.168428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.168660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.168849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.169567] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4812084054, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.169613] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=2, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.176590] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7C-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.176652] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7C-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1156, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.181632] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.184298] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4679598179, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:17.184353] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.187733] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7D-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.187843] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7D-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=982, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.192951] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [18497][T1004_PalfGC][T1004][Y0-0000000000000000-0-0] [lt=12257] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):9216, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):9216, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):563, sequence:1}) [2025-04-03 14:01:17.195636] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077194922]}) [2025-04-03 14:01:17.196313] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660077195687]}) [2025-04-03 14:01:17.196909] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660077196346]}) [2025-04-03 14:01:17.197531] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660077196936]}) [2025-04-03 14:01:17.198686] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660077197946]}) [2025-04-03 14:01:17.199107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.199260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.199418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.199529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.199644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.199746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.199895] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.200160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.200377] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.200543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.200566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.200547] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7E-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.200639] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C7E-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2211, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.200785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.200952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.201374] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.201511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.201662] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.201785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.201937] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.202084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.202237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.202469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.202671] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.202870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.203003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.203144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.209140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2AFE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.209765] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.209882] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=100][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.209932] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.209953] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.215086] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [18228][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=837] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.219847] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.226930] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17722][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=2440] ====== tenant freeze timer task ====== [2025-04-03 14:01:17.231362] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 302.00; ) [2025-04-03 14:01:17.231454] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=60] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:17.231612] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:17.234099] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.234138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.234999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.235092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.235201] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C81-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.237434] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FC-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.248108] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=25][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:17.248156] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=45][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660076047080], now0=[mts=1743660076047080], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:17.248192] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=27][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719469}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660076046171, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:17.248280] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=81][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:17.248319] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=37][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719469}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660076046171, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:17.248418] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=95] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f7dce008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660076045096, tx_desc={this:0x2b7399219100, tx_id:{txid:47719469}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660076046171, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660054039470, use_das=false, nested_level=0, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}, plan=0x2b73976ba050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:17.248521] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=100][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:17.248547] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=23][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:17.248560] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:17.248572] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=9][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:17.248625] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=50][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:17.248647] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=18, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:17.248669] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.257546] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C83-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.262949] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=4812177310, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.262978] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.265056] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.266783] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:18, local_retry_times:18, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:17.266891] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=71][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:17.266908] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:17.266920] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:17.266984] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=37][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:17.267312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.267511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.267644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.267805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.269127] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.269168] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:17.270213] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE377-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.274646] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.280982] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E2-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.281021] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E2-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.281369] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C85-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.281431] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C85-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2559, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.281709] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.281837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E2-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.281988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E2-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.282060] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E2-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.282796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E2-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.283452] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56322 [2025-04-03 14:01:17.284217] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=66][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.285031] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E2-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.294237] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.299561] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56324 [2025-04-03 14:01:17.299849] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.299883] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077299416]}) [2025-04-03 14:01:17.300490] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660077299903]}) [2025-04-03 14:01:17.302665] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56326 [2025-04-03 14:01:17.304414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B00-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.305101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B00-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.305402] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.307884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B00-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.309847] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.309881] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.309891] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.309900] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.316343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B01-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.316453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B01-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.319070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B01-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.319913] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.319916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B01-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.326275] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C89-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.326347] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C89-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1868, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.337156] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8A-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.337199] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8A-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=721, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.344058] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.344083] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56330 [2025-04-03 14:01:17.344130] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56330 [2025-04-03 14:01:17.344138] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.345240] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=41] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:17.345908] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=90] Wash time detail, (compute_wash_size_time=275, refresh_score_time=118, wash_time=19) [2025-04-03 14:01:17.348142] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.348170] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8B-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=806, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.353000] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=7051, clean_start_pos=251656, clean_num=62914) [2025-04-03 14:01:17.358592] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8C-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.358626] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8C-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=348, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.362993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.363216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.363476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.363655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.363896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.364230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.365090] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=51][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.365345] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=162] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.365360] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56332 [2025-04-03 14:01:17.365394] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56332 [2025-04-03 14:01:17.365400] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.365658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.365871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.366041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.366066] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.366194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.366380] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.366550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.366965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.367198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.367395] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.367615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.368030] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4812282586, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.368074] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.369649] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8D-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.369701] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8D-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=852, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.380594] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.380627] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C8E-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=788, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.381765] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.382090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.382940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.383122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.383469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.384054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.384251] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:540) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:1743659476426054000, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=600958193, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="172.20.0.3:2882", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=56820, succ_cluster_heartbeat_count=54701, cluster_heartbeat_interval=1000000, local_cluster_version={val:1743655150322876771, v:0}, local_cluster_delta=4927061371, force_self_check=false, weak_read_refresh_interval=100000) [2025-04-03 14:01:17.384329] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=57] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-04-03 14:01:17.384359] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get_min_unreplayed_log_info(lsn={lsn:28444683855}, scn={val:1743655397585394701, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1743659140625027, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:28444683855}, next_to_submit_scn_:{val:1743655397585394701, v:0}, base_lsn_:{lsn:28318212096}, base_scn_:{val:1743645318682400327, v:0}, iterator_:{iterator_impl:{buf_:0x2b73e2e83000, next_round_pread_size:2121728, curr_read_pos:942632, curr_read_buf_start_pos:0, curr_read_buf_end_pos:942632, log_storage_:{IteratorStorage:{start_lsn:{lsn:28443741223}, end_lsn:{lsn:28444683855}, read_buf:{buf_len_:1167360, buf_:0x2b73e2e83000}, block_size:67104768, log_storage_:0x2b7379685bf0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1743655397585394700, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:452, scn_:{val:1743655397585394700, v:0}, data_checksum:3337333117, flag:0}}, init_mode_version:0, accumulate_checksum:4017040972, curr_entry_is_padding:0, padding_entry_size:484, padding_entry_scn:{val:1743655397585394700, v:0}}}}}) [2025-04-03 14:01:17.384426] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=64] get wrs ts(ls_id={id:1}, delta=4679798392, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:17.384437] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.384518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.384679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.384885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385030] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.385941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386027] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.386967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.387050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.387183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.387284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.387357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.387438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B02-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.399679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B04-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.404346] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.404433] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077403102]}) [2025-04-03 14:01:17.404997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B05-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.405073] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B05-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.405160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B05-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.409052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B05-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.409991] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.410080] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=81][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.410100] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.410116] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.416306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.416364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.416444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.420054] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.420989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.421491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.422951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.423186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.423347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.423518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.423998] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C92-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.424063] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C92-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1001, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.424376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.424462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.424482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.424590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B07-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.426538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.426641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.426749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.426850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.426932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.427154] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.427480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.427567] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.433642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.434467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.434996] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.435720] WDIAG [OCCAM] runTimerTask (ob_occam_timer.h:163) [18437][T1003_MultiVers][T1003][Y0-0000000000000000-0-0] [lt=5818][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.436462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.436655] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C93-0-0] [lt=0][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.436698] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C93-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.436742] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C93-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2424, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.437140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.438540] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B08-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.439157] INFO [STORAGE] runTimerTask (ob_locality_manager.cpp:708) [17536][LocalityReload][T0][Y0-0000000000000000-0-0] [lt=28] runTimer to refresh locality_info(ret=0) [2025-04-03 14:01:17.439541] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:17.439932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B09-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.444070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B09-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.445534] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B09-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.446631] INFO [CLOG] operator() (ob_log_replay_service.cpp:1589) [17804][T1_ReplayProces][T1][Y0-0000000000000000-0-0] [lt=3730] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.447734] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C94-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.449506] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.449528] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56344 [2025-04-03 14:01:17.449568] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56344 [2025-04-03 14:01:17.449923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B0A-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.453017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B0A-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.462998] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] get wrs ts(ls_id={id:1}, delta=4812377913, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.463068] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=63] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.465204] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.469105] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C96-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.469134] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C96-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=861, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.472719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B0B-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.472805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B0B-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.472973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B0B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.475635] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE7D-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.478430] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE7D-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.481907] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.490512] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C98-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.490551] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C98-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=433, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.497950] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:686) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] Cache replace map node details(ret=0, replace_node_count=0, replace_time=3575, replace_start_pos=912253, replace_num=31457) [2025-04-03 14:01:17.498002] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=48] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=10) [2025-04-03 14:01:17.501238] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C99-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.501282] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C99-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=617, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.502295] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:17.502330] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=31] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56346 [2025-04-03 14:01:17.502365] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56346 [2025-04-03 14:01:17.502373] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.505712] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.505745] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56348 [2025-04-03 14:01:17.505781] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56348 [2025-04-03 14:01:17.505792] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.508937] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077507987]}) [2025-04-03 14:01:17.509770] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660077508994]}) [2025-04-03 14:01:17.510179] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=26][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.510420] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660077509833]}) [2025-04-03 14:01:17.511133] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660077510452]}) [2025-04-03 14:01:17.511437] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.511489] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] do_fetch_log_ finished(ret=0, palf_id=1, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:4316157182}, prev_lsn={lsn:4316157060}, fetch_start_log_id=29484365, last_slide_log_id=29484364, fetch_log_size=41943040, accepted_mode_pid=624, last_fetch_req_time=1743660077511487, last_fetch_end_lsn={lsn:4358100222}, last_fetch_max_log_id=29486412, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:29484365, max_lsn:4316157304, max_scn:{val:1743655475525511037, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:4316157304}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:4274214264}}, last_submit_lsn:{lsn:4316157182}, last_submit_end_lsn:{lsn:4316157304}, last_submit_log_id:29484365, last_submit_log_pid:624, max_flushed_lsn:{lsn:4316157182}, max_flushed_end_lsn:{lsn:4316157304}, max_flushed_log_pid:624, committed_end_lsn:{lsn:4316157182}, last_slide_log_id:29484364, last_slide_scn:{val:1743655475423544390, v:0}, last_slide_lsn:{lsn:4316157060}, last_slide_end_lsn:{lsn:4316157182}, last_slide_log_pid:624, last_slide_log_accum_checksum:1508656143, last_fetch_end_lsn:{lsn:4358100222}, last_fetch_max_log_id:29486412, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660077511487, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b736e5ba530}) [2025-04-03 14:01:17.511630] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=132][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.511655] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.512831] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.512870] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.515748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B0C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.520130] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.533704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B0D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.543075] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.543096] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56350 [2025-04-03 14:01:17.543133] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56350 [2025-04-03 14:01:17.543141] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.543152] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0] ====== tenant freeze timer task ====== [2025-04-03 14:01:17.543502] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=23][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.545680] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [17796][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=43564] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:812, free_obj_hold_cnt:138, allocator used:3777200, allocator total:3916800}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:120, free_obj_hold_cnt:2, allocator used:8002102, allocator total:8002102}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=11918902, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:30, free_obj_hold_cnt:19, allocator used:97216, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=932) [2025-04-03 14:01:17.546055] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C9D-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.546162] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C9D-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=910, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.547201] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.547256] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:552) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=49, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=55, Total TxDataTable Memory Limit(MB)=204) [2025-04-03 14:01:17.553589] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:17.553954] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash time detail, (compute_wash_size_time=179, refresh_score_time=81, wash_time=11) [2025-04-03 14:01:17.557331] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C9E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.557409] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C9E-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1050, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.559284] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5288, clean_start_pos=314570, clean_num=62914) [2025-04-03 14:01:17.565023] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.565293] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.566477] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4812480757, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.566528] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.568302] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C9F-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.568380] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3C9F-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=791, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.569281] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [17832][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-04-03 14:01:17.569334] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [17832][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=31] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-04-03 14:01:17.569529] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [17832][T1_ObTimer][T1][Y0-0000000000000000-0-0] [lt=189] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=2374) [2025-04-03 14:01:17.579305] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA0-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.579443] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA0-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=879, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.582103] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=149][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.584488] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4679998115, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:17.584539] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.590666] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA1-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.602721] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.602866] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56352 [2025-04-03 14:01:17.609264] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [18032][T1001_Flush][T1001][Y0-0000000000000000-0-0] [lt=0] ====== traversal_flush timer task ====== [2025-04-03 14:01:17.609905] INFO [MDS] flush (mds_table_mgr.cpp:172) [18032][T1001_Flush][T1001][Y0-0000000000000000-0-0] [lt=70] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b732d454510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743645318682400327, v:0}, clog_base_lsn:{lsn:28318212096}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1743645318682400326, v:0}, all_id_meta:{id_meta:[{limited_id:1743655410707736003, latest_log_ts:{val:1743655394083159448, v:0}}, {limited_id:89000001, latest_log_ts:{val:1743598330496591683, v:0}}, {limited_id:430000001, latest_log_ts:{val:1743620749641273230, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:430, palf_env_:0x2b7320df0030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:759536138, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:17.610077] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [18032][T1001_Flush][T1001][Y0-0000000000000000-0-0] [lt=161] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-04-03 14:01:17.610176] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.610217] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.610229] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.610237] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.613514] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077612789]}) [2025-04-03 14:01:17.613932] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA3-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.614046] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA3-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=847, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.614312] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660077613583]}) [2025-04-03 14:01:17.615226] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660077614389]}) [2025-04-03 14:01:17.615947] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660077615263]}) [2025-04-03 14:01:17.618118] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.620247] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.622738] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56354 [2025-04-03 14:01:17.636562] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA5-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.636646] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA5-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1364, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.639576] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.639597] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56356 [2025-04-03 14:01:17.639636] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56356 [2025-04-03 14:01:17.639643] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.642431] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.642444] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56358 [2025-04-03 14:01:17.642587] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56358 [2025-04-03 14:01:17.642596] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.645412] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=57] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:17.647128] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.647144] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56360 [2025-04-03 14:01:17.647186] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56360 [2025-04-03 14:01:17.647194] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.648419] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA6-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.648544] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA6-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1483, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.659724] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA7-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.659803] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA7-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=900, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.662726] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.663177] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=4812578057, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.663228] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=43] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.665381] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=54][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.666028] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.666050] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56362 [2025-04-03 14:01:17.666095] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56362 [2025-04-03 14:01:17.666104] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.671380] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA8-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.671462] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CA8-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1378, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.672390] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.682173] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.685093] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.695676] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAA-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.695745] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAA-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2061, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.698948] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=9) [2025-04-03 14:01:17.702902] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.702921] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56364 [2025-04-03 14:01:17.702967] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56364 [2025-04-03 14:01:17.705939] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:17.706763] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAB-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.710246] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.710356] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=102][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.710386] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] re_register_parent reason(palf_id=1001, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660073706268, last_first_register_time_us=1743648246815801, register_time_us=1743660073706268, curr_time_us=1743660077710384) [2025-04-03 14:01:17.711635] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.711668] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.717168] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=56] ====== check clog disk timer task ====== [2025-04-03 14:01:17.717210] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=30] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=315) [2025-04-03 14:01:17.717350] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAC-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.717392] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAC-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=446, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.718251] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=18] cannot_recycle_log_size statistics(cannot_recycle_log_size=129898528, threshold=483183820, need_update_checkpoint_scn=false) [2025-04-03 14:01:17.720311] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.721684] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077718291]}) [2025-04-03 14:01:17.723001] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660077721737]}) [2025-04-03 14:01:17.723667] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660077723073]}) [2025-04-03 14:01:17.724242] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660077723696]}) [2025-04-03 14:01:17.725302] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660077724635]}) [2025-04-03 14:01:17.730518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B11-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.739186] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAE-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.739257] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAE-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1050, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.743523] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:17.746276] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.749292] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=49] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1003}, {id:1002}]) [2025-04-03 14:01:17.749435] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.749471] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56370 [2025-04-03 14:01:17.749555] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56370 [2025-04-03 14:01:17.749592] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.751352] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAF-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.751426] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CAF-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1821, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.758007] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0] ====== tenant freeze timer task ====== [2025-04-03 14:01:17.758428] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.759451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.759769] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:17.760125] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash time detail, (compute_wash_size_time=165, refresh_score_time=81, wash_time=11) [2025-04-03 14:01:17.762380] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB0-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.762489] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB0-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=895, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.763294] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.763983] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=3837, clean_start_pos=377484, clean_num=62914) [2025-04-03 14:01:17.765458] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=57][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.766677] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4812681723, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.766726] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.768075] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:17.768106] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.768140] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.768128] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=53][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:19072, tid:17828}]) [2025-04-03 14:01:17.768170] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:552) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=512, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=33, Freeze TxData Trigger Memory(MB)=10, Total TxDataTable Hold Memory(MB)=36, Total TxDataTable Memory Limit(MB)=51) [2025-04-03 14:01:17.773738] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB1-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.773829] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB1-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1156, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.782265] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.784647] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4680198807, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:17.784689] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.785214] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB2-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.785280] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB2-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1156, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.789147] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.789181] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=26] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56372 [2025-04-03 14:01:17.789269] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56372 [2025-04-03 14:01:17.789286] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.796101] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB3-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.796140] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB3-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=563, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.806829] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB4-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.806859] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB4-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=592, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.808625] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=56] PNIO [ratelimit] time: 1743660077808621, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-04-03 14:01:17.810366] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.810385] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.810392] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.810400] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.817225] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB5-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.817252] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB5-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=291, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.820387] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.822182] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:87) [17994][T1001_ReqMemEvi][T1001][Y0-0000000000000000-0-0] [lt=1] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-04-03 14:01:17.823331] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=40][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.823399] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=2][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.823417] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.823503] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1098) [18115][T1002_TabletSta][T1002][Y0-0000000000000000-0-0] [lt=11180] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1002, interval_step=1, last_update_time=1743660077823502) [2025-04-03 14:01:17.823624] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.823653] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=28][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.823733] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.823757] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.823862] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.823881] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.823954] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824025] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=70][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824104] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824115] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824165] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824173] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824235] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824255] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824327] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824346] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824419] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824455] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=35][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824565] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824584] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.824672] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824683] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824757] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824769] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824845] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824855] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.824929] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.824948] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.825488] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.825520] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=31][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.825639] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.825663] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.825779] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.825793] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:17.825932] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:17.826759] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077826046]}) [2025-04-03 14:01:17.827466] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660077826865]}) [2025-04-03 14:01:17.827854] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17994][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6BD5CBE41-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.828286] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660077827508]}) [2025-04-03 14:01:17.829108] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6C9-0-0] [lt=1098][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.829052] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB6-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.829138] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB6-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1780, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.831564] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.843625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B13-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.846650] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:17.846688] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=31] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56376 [2025-04-03 14:01:17.846803] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56376 [2025-04-03 14:01:17.847116] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [17564][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=34] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.849845] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17560][BatchIO][T0][Y0-0000000000000000-0-0] [lt=2] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.851007] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=62] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.851302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.851465] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.851571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.851801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.851878] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.852057] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.852231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.852393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.852583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.852737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.853082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.853214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.853376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.853549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.853749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.853869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.854043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B14-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.862438] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB9-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.862484] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CB9-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=313, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.863325] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=4812778032, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:17.863352] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.865859] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=63][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.866932] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:17.866978] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56378 [2025-04-03 14:01:17.867073] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56378 [2025-04-03 14:01:17.867101] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:17.872824] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBA-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.872855] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBA-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=302, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.873523] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18296][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=412][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.882351] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.883522] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBB-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.883566] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBB-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=567, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.884712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.884734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.884838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.884862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.884980] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885586] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.885997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886051] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886159] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886204] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.886328] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-04-03 14:01:17.886377] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=4680300545, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:17.886401] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:17.894748] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBC-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.894832] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBC-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1094, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.899094] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=36] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=8) [2025-04-03 14:01:17.905854] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBD-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.905955] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBD-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=910, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.910444] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.910504] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=53][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.910525] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.910548] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.916964] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBE-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.917077] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBE-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=928, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.920514] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:17.923426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=4] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.923455] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.923693] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.923711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.923897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.923914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924094] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924763] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.924988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.925290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.925310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.925494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.925510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.925723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.925738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.925933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.925970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.926417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.926435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.926596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.926611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.926933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.926951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.927160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.927176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.927580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.927597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.928038] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.928056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.928272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.928287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.928647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.928665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.929056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.929074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.929569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.929585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.929978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.929994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.930260] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBF-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.930315] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CBF-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=3027, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.930453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.930469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.930703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.930737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.930998] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.931032] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.931253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.931270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.931522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.931537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.931672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.931707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.932028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.932044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.932356] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.932395] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=37] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.932614] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.932630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.932854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.932869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.933949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.934178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.934195] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.934361] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.934375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.934604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.934620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.934875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.934890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.935269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.935285] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.935683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.935698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.936085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.936101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.936412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.936426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.936685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.936706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.936885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.936894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937021] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660077934971]}) [2025-04-03 14:01:17.937129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937551] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937950] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.937960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938174] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938293] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660077937043]}) [2025-04-03 14:01:17.938404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938586] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.938984] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.939320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.939330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.939544] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.939555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.939727] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660077938311]}) [2025-04-03 14:01:17.939788] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.939797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.940000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.940487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=485] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.941493] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC0-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.941529] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC0-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=991, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.941758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.941771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.942080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.942092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.942159] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660077939742]}) [2025-04-03 14:01:17.942517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.942547] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.942792] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.942803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.943042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.943053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.943342] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.943354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.943640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.943652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.943918] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660077942394]}) [2025-04-03 14:01:17.944363] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.944400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=36] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.944584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.944775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.945075] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.945252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.945589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.946053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.946206] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.946380] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.946408] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:261) [17837][T1_ArcSender][T1][YB42AC140005-000631D6BB2CB54C-0-0] [lt=61][errcode=-4018] try free send task failed(ret=-4018) [2025-04-03 14:01:17.946521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.946876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B15-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.947470] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [17796][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=78] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:812, free_obj_hold_cnt:138, allocator used:3777200, allocator total:3916800}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:120, free_obj_hold_cnt:2, allocator used:8002102, allocator total:8002102}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=11918902, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:30, free_obj_hold_cnt:19, allocator used:97216, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=932) [2025-04-03 14:01:17.948207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.948239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.948308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.948486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.949028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.949137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.949305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.949459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.949780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.949942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.950262] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.950673] WARN alloc_chunk (achunk_mgr.cpp:326) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit(msg="oops, over total memory limit", hold=10735362048, limit=10737418240) [2025-04-03 14:01:17.950770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.951109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.951242] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.951567] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.951920] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.952581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.952733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.953061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.953685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.954049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.954189] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.954333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B16-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.955558] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.955586] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.955887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.956069] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.956230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.956426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.956509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.956677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.957114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.957265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.957606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.957759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.958243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.958406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.958768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.958936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.959086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.959576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.959720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.960052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.960200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.960719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.960871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.961031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B17-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.961848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.961857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.962101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.962188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.962483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.962645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.962823] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.963156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.963268] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC2-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.963333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.963475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.963798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.963948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.964108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.964292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.964445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.964595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.964633] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.964922] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.965081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.965442] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.965591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.965901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.965933] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.965957] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:17.966092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.966225] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.966408] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.966744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B18-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.967501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.967513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.967693] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.967851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.968028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.968346] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.968424] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4658, tid:17484}]) [2025-04-03 14:01:17.968523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.968714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.969034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.969175] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.969343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.969486] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.969494] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=8][errcode=-4013] rpc memory pool alloc memory failed(sz=1374, alloc_sz=16256) [2025-04-03 14:01:17.969501] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1190) [2025-04-03 14:01:17.969506] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=5][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.969514] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1003, svr="172.20.0.3:2882", timeout=998674, timeout_timestamp=1743660078966942) [2025-04-03 14:01:17.969529] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=14][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:17.969535] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=5][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:17.969540] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=5][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:17.969545] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=5][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:17.969550] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=4][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:17.969566] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=15][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:17.969574] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=7][errcode=-4013] execute failed(ret=-4013, tenant_id=1003, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:17.969582] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=7][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:17.969607] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=15][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.969612] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=5][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.969616] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=4][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:17.969731] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.969738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.969851] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=29] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1}]) [2025-04-03 14:01:17.969966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.970133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.970295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.970459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.970622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.970851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.970969] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.971107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.971286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.971434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.971787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.971929] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.972134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.972445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.972778] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.972926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.973223] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:17.973231] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B19-0-0] [lt=7][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:17.973240] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=7][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:17.973244] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=3][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.973250] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=5][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1003) [2025-04-03 14:01:17.973254] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=4][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1003) [2025-04-03 14:01:17.973258] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=4][errcode=-4013] query failed(ret=-4013, conn=0x2b72e59ac050, start=1743660077966947, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:17.973266] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=7][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.973315] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=14][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004) [2025-04-03 14:01:17.973320] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=5][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:17.973328] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=8][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:17.973333] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=5][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1004, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660077943913}) [2025-04-03 14:01:17.973345] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=6][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:17.973349] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B19-0-0] [lt=3][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:17.974331] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.974343] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=11][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:17.974348] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=4][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:17.974353] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.974360] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=4][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:17.974393] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=32] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.974405] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=12][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.974417] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=12] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=913, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.974429] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC3-0-0] [lt=11][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.974624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.974632] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.974682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.974687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.974747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.974886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.975024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.975232] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.975362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.975540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.975666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.975840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.976022] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.976171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.976299] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.976306] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=6][errcode=-4013] rpc memory pool alloc memory failed(sz=1341, alloc_sz=16256) [2025-04-03 14:01:17.976310] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=4][errcode=-4013] alloc buffer fail(payload_sz=1157) [2025-04-03 14:01:17.976314] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.976320] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=998258, timeout_timestamp=1743660078973549) [2025-04-03 14:01:17.976327] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=6][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:17.976331] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:17.976334] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:17.976338] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:17.976351] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=12][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:17.976354] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:17.976360] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=5][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:17.976366] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=5][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:17.976380] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=10][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.976383] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.976386] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:17.976513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.976652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.976866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.976989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.977135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.977334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.977481] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.977635] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.977800] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.977958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.978171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.978294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.978450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.978657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.978763] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.978924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.979261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.979437] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.979585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.979698] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:17.979705] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=6][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:17.979712] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=5][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:17.979716] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.979721] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=4][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:17.979724] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:17.979728] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] query failed(ret=-4013, conn=0x2b73a1788050, start=1743660077973554, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:17.979744] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=14][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.979775] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=5][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:17.979780] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=4][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1002, ls_ids=[{id:1001}]) [2025-04-03 14:01:17.979786] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=5][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1002, ls_ids=[{id:1001}]) [2025-04-03 14:01:17.979790] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=4][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1002, ls_id:{id:1001}, renew_for_tenant:false, add_timestamp:1743660077950114}) [2025-04-03 14:01:17.979796] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=6][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:17.979800] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1A-0-0] [lt=3][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:17.980397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.980407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.980606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.980732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.980892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.981230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.981540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.981706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.981867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.981966] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.981974] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=8][errcode=-4013] rpc memory pool alloc memory failed(sz=1375, alloc_sz=16256) [2025-04-03 14:01:17.981979] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=4][errcode=-4013] alloc buffer fail(payload_sz=1191) [2025-04-03 14:01:17.981983] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.981991] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1003, svr="172.20.0.3:2882", timeout=998799, timeout_timestamp=1743660078979866) [2025-04-03 14:01:17.981997] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=6][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:17.982031] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=32][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:17.982035] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:17.982039] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=4][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:17.982043] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=3][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:17.982047] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=3][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:17.982053] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=5][errcode=-4013] execute failed(ret=-4013, tenant_id=1003, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:17.982060] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=6][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:17.982075] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=11][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.982078] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=3][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.982082] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=3][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:17.982210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.982224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.982372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.982491] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=53][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:17.982570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.982688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.982857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.982993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.983191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.983319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.983509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.983692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.983832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.984020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.984156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.984321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.984652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.984799] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=4680398987, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:17.985156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.985271] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC4-0-0] [lt=20][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.985285] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC4-0-0] [lt=13][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:17.985289] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC4-0-0] [lt=4][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:17.985293] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC4-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.985300] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC4-0-0] [lt=4][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:17.985317] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC4-0-0] [lt=17] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.985339] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC4-0-0] [lt=21][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.985351] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC4-0-0] [lt=0][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.985435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.985833] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.985957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.986225] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:17.986238] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=12][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:17.986254] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=15][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:17.986259] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=4][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.986263] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=4][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1003) [2025-04-03 14:01:17.986267] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=3][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1003) [2025-04-03 14:01:17.986270] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=3][errcode=-4013] query failed(ret=-4013, conn=0x2b738fe60050, start=1743660077979870, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:17.986277] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=5][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1001) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.986306] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=4][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004) [2025-04-03 14:01:17.986313] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=7][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1001}]) [2025-04-03 14:01:17.986320] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=6][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1001}]) [2025-04-03 14:01:17.986335] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=15][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1004, ls_id:{id:1001}, renew_for_tenant:false, add_timestamp:1743660077949915}) [2025-04-03 14:01:17.986341] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=6][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:17.986345] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1B-0-0] [lt=3][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:17.986965] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=3][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:17.986985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.986991] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.987043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.987087] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.987237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.987431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.987687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.987755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.987899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.988162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=57] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.988218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.988424] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.988560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.988692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.988712] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.988719] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=7][errcode=-4013] rpc memory pool alloc memory failed(sz=1375, alloc_sz=16256) [2025-04-03 14:01:17.988724] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=5][errcode=-4013] alloc buffer fail(payload_sz=1191) [2025-04-03 14:01:17.988729] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=5][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.988737] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1003, svr="172.20.0.3:2882", timeout=998755, timeout_timestamp=1743660078986371) [2025-04-03 14:01:17.988743] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=5][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:17.988764] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=20][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:17.988770] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=5][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:17.988774] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=3][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:17.988777] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=2][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:17.988781] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=3][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1003) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:17.988787] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=6][errcode=-4013] execute failed(ret=-4013, tenant_id=1003, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1003) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:17.988794] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=6][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:17.988818] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=10][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.988822] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=3][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:17.988825] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=2][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:17.988864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.989052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.989208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.989344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.989596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.989697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.989839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.990031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.990161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.990541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.990659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.990829] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.990976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.991320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.991630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.991842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.991973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.992140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.992291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:17.992575] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:17.992587] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=10][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1003) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:17.992595] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=6][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1003) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:17.992600] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=4][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1003) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.992614] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=14][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1003) [2025-04-03 14:01:17.992619] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=4][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1003) [2025-04-03 14:01:17.992623] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=3][errcode=-4013] query failed(ret=-4013, conn=0x2b7322070050, start=1743660077986375, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1003) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:17.992630] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=5][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1003) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:17.992663] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=5][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004) [2025-04-03 14:01:17.992673] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=9][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1003}]) [2025-04-03 14:01:17.992681] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=7][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1003}]) [2025-04-03 14:01:17.992686] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=4][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1004, ls_id:{id:1003}, renew_for_tenant:false, add_timestamp:1743660077954974}) [2025-04-03 14:01:17.992692] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=6][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:17.992696] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B1C-0-0] [lt=3][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:17.995832] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=12][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:17.995858] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=25][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:17.995863] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=4][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:17.995868] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:17.995874] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=3][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:17.995893] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=19] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:17.995905] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=12][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.995918] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=12] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=468, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:17.995944] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC5-0-0] [lt=25][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.006304] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC6-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.006334] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC6-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=282, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.009692] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [17844][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:18.010517] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.010538] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.010546] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.010564] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.016889] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC7-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.016942] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC7-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=499, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.020586] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=43][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.026216] INFO [ARCHIVE] get_send_task_ (ob_archive_sender.cpp:364) [18639][T1004_ArcSender][T1004][YB42AC140005-000631D6C0ECB54C-0-0] [lt=15] task not exist, wait time(time_cost=100362) [2025-04-03 14:01:18.027675] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC8-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.027731] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC8-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=655, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.034082] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.034156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=69] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034174] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034750] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.034967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035154] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=42] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.035992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036206] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=40] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.036837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=55] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037181] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037552] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.037684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.037885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038002] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038258] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038401] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.038777] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC9-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.038905] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CC9-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=985, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.039102] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.039122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.039246] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.039265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.039373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.039393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.039900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.039930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.040097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.040122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.040213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.040236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.040603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.040632] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.041321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.041348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.041477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.041497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.041612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.041660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=46] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.041857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.041880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042823] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.042992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043021] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043558] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043917] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.043936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.044073] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.044092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.044499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.044527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.044738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.044777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=37] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.045189] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.045214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.045250] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660078044374]}) [2025-04-03 14:01:18.045566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.045650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=62] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.045863] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660078045300]}) [2025-04-03 14:01:18.046134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.046164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.046288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.046335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=45] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.046469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.046493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.046610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.046633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=44] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047195] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047219] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047766] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660078045914]}) [2025-04-03 14:01:18.047863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.047999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048251] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048663] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.048900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049276] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049547] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049663] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.049974] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.050194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.050473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.050636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.050764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.050928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.051087] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.051250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.051413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.051593] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.051745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.051910] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.052064] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.052230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.052390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.052550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.052721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.052878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.053048] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.053202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.053366] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.053529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.053701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.053889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.054055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.054218] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.054529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.054674] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.055032] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B1D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.055498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.055513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.055839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.055851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.055987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.056221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.056346] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.056462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.056636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.056787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.056951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.057189] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.057302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.057479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.057601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.057774] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.057931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.058092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18630][T1004_ArcSrv][T1003][YB42AC140005-000631D6C0ACB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.058420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.058598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.058768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.058938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.059068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.059255] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.059393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.059556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.059717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.059896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.060043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.060236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.060381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.060548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.060710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.060871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.061046] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.061386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.062212] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.062229] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.062280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.062253] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CCB-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.062333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.062517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.062670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.062820] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.063022] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.063150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.063494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.063660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.063834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.063952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.064282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.064447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.064830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.064930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.065104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.065316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.065461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.065747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.065921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.066028] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.066071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.066230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.066403] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.066570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.066765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.066907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.066926] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=606] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.067043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.067208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.067372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.067545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.067732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B1F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.067894] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=0] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2b73c068a510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616808737813004, v:0}, clog_base_lsn:{lsn:4697333760}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743622739127262315, v:0}, all_id_meta:{id_meta:[{limited_id:1743655270710910399, latest_log_ts:{val:1743655254062460505, v:0}}, {limited_id:35000001, latest_log_ts:{val:1743639632198700904, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:450, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:18.068001] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:4697333760}, ls->get_ls_id()={id:1}) [2025-04-03 14:01:18.068613] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=15][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:9939, tid:18224}]) [2025-04-03 14:01:18.068823] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=24] succ to get rec scn(*this={this:0x2b7346604030, ls_id:{id:1001}, tenant_id:1002, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1743648246000622, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-04-03 14:01:18.068843] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=19] tx ctx memtable get rec scn(this={ObITable:{this:0x2b73484fdca0, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b73484fdca0, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-04-03 14:01:18.068907] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1634) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}, curr_base_lsn={lsn:3690762240}, new_base_lsn={lsn:3690762240}, lsn={lsn:3690762240}) [2025-04-03 14:01:18.068917] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:3690762240}, ls->get_ls_id()={id:1001}) [2025-04-03 14:01:18.069741] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1412) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=5] succ to get rec scn(*this={this:0x2b7397804030, ls_id:{id:1002}, tenant_id:1002, state:"F_WORKING", total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1743648246234932, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:3}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-04-03 14:01:18.069756] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:180) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=14] tx ctx memtable get rec scn(this={ObITable:{this:0x2b73484fce90, key:{tablet_id:{id:49401}, column_group_idx:0, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x2b73484fce90, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1002}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-04-03 14:01:18.069792] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:791) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=32] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:18446744073709551615, v:3}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:18446744073709551615, v:3}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1002}) [2025-04-03 14:01:18.069837] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:259) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=12] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x2b73b7f94510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1743655484668882629, v:0}, clog_base_lsn:{lsn:3556552704}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743648258592483359, v:0}, all_id_meta:{id_meta:[{limited_id:1743494462089433682, latest_log_ts:{val:1743494445468388405, v:0}}, {limited_id:33000001, latest_log_ts:{val:1743494442089433683, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:171, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:18.069888] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1634) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}, curr_base_lsn={lsn:3556552704}, new_base_lsn={lsn:3556552704}, lsn={lsn:3556552704}) [2025-04-03 14:01:18.069898] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [18236][T1002_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=10] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:3556552704}, ls->get_ls_id()={id:1002}) [2025-04-03 14:01:18.070947] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.073111] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CCC-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.073162] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CCC-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=777, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.081641] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.081677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.081690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.081699] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=9][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.081782] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.081797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.081825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082024] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.082044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082051] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082133] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.082157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082243] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.082257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082342] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.082361] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082366] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082464] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.082476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082575] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.082609] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.082619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082723] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.082740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082866] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.082878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.082884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083065] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.083190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=123] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083379] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.083393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083491] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.083516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083627] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.083638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083662] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083754] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.083766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083772] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.083877] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CCD-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.083904] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CCD-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=637, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.083978] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.084114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=134] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.084122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.084303] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.084318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.084326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.084440] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:18.084470] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.084847] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.084861] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.084900] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.084930] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.084950] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.084971] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.084977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.085126] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.085350] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.085361] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.086227] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.086250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.086544] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.086565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.086445] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.086692] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.086762] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.086875] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.086999] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=123] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.087034] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.087027] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.087053] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.087128] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.087176] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.087267] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.087279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.087383] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.087592] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.087633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.087771] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.089481] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.089606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=123] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.089617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.089748] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.089975] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.089997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.090335] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.090365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.090489] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.090651] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.090786] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.090976] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.091104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=126] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.091294] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.091309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.091369] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.091566] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.091600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.091729] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.091910] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.091944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.092065] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.092231] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.092492] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.092620] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.092646] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.092748] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.092867] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.092996] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.093021] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.093147] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.093283] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.093295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.093460] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.093578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=116] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.093917] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.093930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.094223] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CCE-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.094041] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.094477] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.094655] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.094682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=26] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.094783] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.094911] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.094935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.095126] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.095248] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.095823] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=554] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.095840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.096141] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.096249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=106] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.096509] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.096524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.096684] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.096713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.097001] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.097034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.097209] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.097373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=162] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.097560] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.097576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.097755] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.098096] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.098113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.098121] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.098297] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.098309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.098436] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.098636] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.098648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.098805] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.098912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=106] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.099071] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.099201] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=7) [2025-04-03 14:01:18.099302] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.099328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.099409] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.099573] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.099673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=99] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.099790] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.099928] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.099955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.100126] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.100313] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.100718] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.100733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.100915] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.101090] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.101102] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.101184] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.101348] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.101497] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.101681] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.101691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.101796] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:18.101921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.102100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.102244] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.102591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.103057] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.103235] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.103425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2D9-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.105590] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CCF-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.105646] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CCF-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1039, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.110618] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.110673] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.110697] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.110725] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.117400] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD0-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.117451] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD0-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1671, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.117503] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1980) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=1] dump_dag_status(dag_cnt=0, map_size=0) [2025-04-03 14:01:18.117521] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1990) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(running_dag_net_map_size=0, blocking_dag_net_list_size=0) [2025-04-03 14:01:18.117545] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(priority="PRIO_COMPACTION_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117557] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_HIGH", low_limit=8, up_limit=8, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117563] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_COMPACTION_MID", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117568] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_HA_MID", low_limit=5, up_limit=5, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117573] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_COMPACTION_LOW", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117579] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_HA_LOW", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117584] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_DDL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117589] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=4] dump_dag_status(priority="PRIO_DDL_HIGH", low_limit=6, up_limit=6, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117594] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:1999) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=5] dump_dag_status(priority="PRIO_TTL", low_limit=2, up_limit=2, running_task=0, ready_dag_count=0, waiting_dag_count=0) [2025-04-03 14:01:18.117601] INFO [COMMON] dump_dag_status (ob_dag_scheduler.cpp:2014) [18573][T1004_DagSchedu][T1004][Y0-0000000000000000-0-0] [lt=6] dump_dag_status(total_worker_cnt=43, total_running_task_cnt=0, work_thread_num=43, scheduled_task_cnt=0) [2025-04-03 14:01:18.120905] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=235][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.127266] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.127338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=68] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.127358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.127927] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.127944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.128498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.128516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.128770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.128783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.129054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.129068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.129095] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD1-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.129333] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD1-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1729, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.129481] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.129495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.129737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.129751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.130122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.130266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=142] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.130721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.130737] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.131365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.131382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.131835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.131858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.132092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.132103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.132473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.132495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.132662] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=36][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:18.132698] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=35][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660105653097, now=[mts=1743660076927939], now0=[mts=1743660076927939], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:18.132736] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=35][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a431d2a0, tx_id:{txid:76175013}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660076919118, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:18.132831] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=87][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b739ba5c1a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a431d2a0}) [2025-04-03 14:01:18.132869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.132882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.132873] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=40][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a431d2a0, tx_id:{txid:76175013}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660076919118, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:18.132933] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=56] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f8054008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660076925667, tx_desc={this:0x2b73a431d2a0, tx_id:{txid:76175013}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660076919118, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660075723113, use_das=false, nested_level=0, session={this:0x2b739ba5c1a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a431d2a0}, plan=0x2b7338b48050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:18.133034] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=99][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:18.133055] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=18][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:18.133068] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=11][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:18.133080] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=10][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:18.133091] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=9][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:18.133126] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=32][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=1, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:18.133144] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=13] will sleep(sleep_us=1000, remain_us=27589954, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=1, v.err_=-4283, timeout_timestamp=1743660105723097) [2025-04-03 14:01:18.133169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.133179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.133459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.133480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.133928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.133949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134227] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=15][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:1, local_retry_times:1, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:18.134284] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=40][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:18.134293] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=8][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:18.134300] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=7][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:18.134371] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=9][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.134399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134646] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134778] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.134866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.135090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.135287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.135430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.135596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.135768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.135939] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.136269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.136384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.136543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.136755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.136873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.137029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.137190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.137353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.137702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.138056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.138176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.138513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.138669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.138865] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.139166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.139367] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.139658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.139966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.140203] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.140328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.140398] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.140432] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:18.140669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.141127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.141316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.141625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.141753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.142443] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:18.142960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.143263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154C-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.151455] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=42] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.159158] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660078158301]}) [2025-04-03 14:01:18.159656] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660078159199]}) [2025-04-03 14:01:18.160740] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660078160087]}) [2025-04-03 14:01:18.160911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B20-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.161208] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:110) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=74] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.166102] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.173665] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=56] ====== tenant freeze timer task ====== [2025-04-03 14:01:18.182846] INFO [PALF] log_loop_ (log_loop_thread.cpp:175) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23] LogLoopThread round_cost_time(us)(round_cost_time=42) [2025-04-03 14:01:18.182866] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.184923] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4680599279, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:18.184956] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.186137] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD6-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.186204] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD6-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=985, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.197511] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD7-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.197732] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD7-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1363, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.204553] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.204649] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=88] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56380 [2025-04-03 14:01:18.204746] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56380 [2025-04-03 14:01:18.204771] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.208881] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD8-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.208957] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD8-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=893, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.210697] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.210757] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.210776] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.210794] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.220752] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.220836] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD9-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.220945] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CD9-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1760, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.231441] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CDA-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.231490] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CDA-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=396, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.232989] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=51] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 773.00; ) [2025-04-03 14:01:18.233055] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=40] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:18.233182] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:18.239799] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.239884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.240029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.240160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.240365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.240496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.240638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.240784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.240926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.241067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.241268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.241520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.241662] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.241789] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.241932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.242074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.242196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.242400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.242528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.242509] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CDB-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.242559] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CDB-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=839, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.242749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.242934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.243109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.243349] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.243487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.243747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.244217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.244354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.244473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.244596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.244721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.244863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.245042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.245200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.245483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.245721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.245859] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.245994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.246173] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.246391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.246961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.247029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.247131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.247311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.247380] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FD-0-0] [lt=18][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.247445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.247569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.247836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.247986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.248136] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.248459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.248628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.248842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.248980] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.249343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B22-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.262302] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660078261451]}) [2025-04-03 14:01:18.263580] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4813178377, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:18.263569] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660078262384]}) [2025-04-03 14:01:18.263606] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.266183] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=59][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.266280] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CDD-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.268638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B23-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.271758] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8CB-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.275273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B23-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.276968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B23-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.278919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B23-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.279600] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE378-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.279669] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B23-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.282948] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.283654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B23-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.284995] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.285029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.285046] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8CB-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.286367] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.287153] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8CB-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.288951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17937][T1001_PlanCache][T1001][YB42AC140005-000631D6C60CC8CB-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.289599] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E3-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.289968] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CDF-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.295394] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E3-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.299301] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.300397] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE0-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.300433] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE0-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=299, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.302430] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=9] ====== check clog disk timer task ====== [2025-04-03 14:01:18.302455] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=1024, used(MB):=255) [2025-04-03 14:01:18.302479] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=7] cannot_recycle_log_size statistics(cannot_recycle_log_size=200526433, threshold=322122547, need_update_checkpoint_scn=false) [2025-04-03 14:01:18.310789] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.310858] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=61][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.310882] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.310900] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.311312] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE1-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.311352] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE1-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=801, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.320842] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.322168] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE2-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.322226] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE2-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=588, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.333224] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE3-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.333323] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE3-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=908, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.344029] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:18.344054] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56384 [2025-04-03 14:01:18.344090] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56384 [2025-04-03 14:01:18.344107] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.344293] INFO [ARCHIVE] get_send_task_ (ob_archive_sender.cpp:364) [18637][T1004_ArcSender][T1004][YB42AC140005-000631D6C0DCB54C-0-0] [lt=679] task not exist, wait time(time_cost=100468) [2025-04-03 14:01:18.346276] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE4-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.346346] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE4-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2711, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.349097] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [17796][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=60] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:812, free_obj_hold_cnt:138, allocator used:3777200, allocator total:3916800}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:120, free_obj_hold_cnt:2, allocator used:8002102, allocator total:8002102}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=11918902, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:30, free_obj_hold_cnt:19, allocator used:97216, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=932) [2025-04-03 14:01:18.357482] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE5-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.357617] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE5-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1072, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.364680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.365076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.365388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.365625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.366297] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.366401] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.366701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.366877] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.367049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.367167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.367370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.367518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.367703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.367832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.367965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.368166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.368202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.368330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.368462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.368590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.368791] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.368921] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660078367145]}) [2025-04-03 14:01:18.368944] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4813283196, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:18.368968] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.369896] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.369939] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56386 [2025-04-03 14:01:18.370115] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56386 [2025-04-03 14:01:18.370154] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.371034] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE6-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.371095] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE6-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=3224, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.371789] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660078368958]}) [2025-04-03 14:01:18.375108] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660078371945]}) [2025-04-03 14:01:18.376964] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660078375139]}) [2025-04-03 14:01:18.378864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.379099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.379279] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660078377645]}) [2025-04-03 14:01:18.379373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.379410] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=1] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:18.379501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.379648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.379876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.380003] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.380099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.380215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.380319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.380388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.380424] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=87] Wash time detail, (compute_wash_size_time=513, refresh_score_time=252, wash_time=32) [2025-04-03 14:01:18.380898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.381179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.382411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.383112] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.383249] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE7-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.383306] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE7-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1790, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.383471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.383728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.383932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.384124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.384446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.384719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.384886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.385026] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.385131] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4680799275, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:18.385153] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.385892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.386312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.386514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.386774] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.386935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.387091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.387247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.387597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.387734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.387837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.387966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.388170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.388293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.388418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.388677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.388871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.389078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.389222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.389722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.390053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.390209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.390610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.391271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.391421] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=66] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=10925, clean_start_pos=566226, clean_num=62914) [2025-04-03 14:01:18.391865] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.392318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.393044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.393324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.393647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.394275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.394362] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE8-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.394391] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CE8-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=919, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.394491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.394715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.394997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B25-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.404934] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56388 [2025-04-03 14:01:18.410839] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.410868] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.410887] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.410898] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.415707] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CEA-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.415755] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CEA-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=811, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.420921] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.426311] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CEB-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.426347] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CEB-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=455, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.437110] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CEC-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.437155] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CEC-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=668, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.440185] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [18401][T1003_LockWaitM][T1003][Y0-0000000000000000-0-0] [lt=46] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-04-03 14:01:18.444837] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:18.444876] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=31] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56390 [2025-04-03 14:01:18.444973] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56390 [2025-04-03 14:01:18.445022] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.448076] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CED-0-0] [lt=1][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.448127] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CED-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.448188] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CED-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=871, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.453334] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18111][T1002_LogUpdate][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.459677] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CEE-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.459773] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CEE-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1352, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.464000] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69] get wrs ts(ls_id={id:1}, delta=4813378656, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:18.464071] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=64] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.464404] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B27-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.466380] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=106][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.472186] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:18.472232] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=44][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660077267218], now0=[mts=1743660077267218], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:18.472270] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=34][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719470}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660077266623, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:18.472409] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=130][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:18.472459] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=48][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719470}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660077266623, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:18.472521] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=57] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f7dce008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660077257871, tx_desc={this:0x2b7399219100, tx_id:{txid:47719470}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660077266623, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660054039470, use_das=false, nested_level=0, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}, plan=0x2b73976ba050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:18.472647] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=123][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:18.472678] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=28][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:18.472697] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=17][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:18.472716] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:18.472737] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=18][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:18.472781] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=41][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=19, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:18.472855] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=68] will sleep(sleep_us=19000, remain_us=5566606, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=19, v.err_=-4283, timeout_timestamp=1743660084039460) [2025-04-03 14:01:18.483110] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=66][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.483171] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=60][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.486529] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660078485995]}) [2025-04-03 14:01:18.487244] ERROR issue_dba_error (ob_log.cpp:1875) [18436][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4013, file="ob_multi_version_garbage_collector.cpp", line_no=222, info="repeat refresh failed too much time") [2025-04-03 14:01:18.487264] EDIAG [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:222) [18436][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] repeat refresh failed too much time(ret=-4013, this={this:0x2b7349bc71a0, last_study_timestamp:1743654917656496, last_refresh_timestamp:1743655217693956, last_reclaim_timestamp:1743659778558486, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:true, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:1743654794549130320, v:0}, is_inited:true}, current_timestamp=1743660078454884) BACKTRACE:0x125fd785 0x50b6335 0x51fab28 0x51fa63b 0x51fa57e 0x51fa3a6 0xef83d9e 0xef927e9 0x5209517 0x5206c75 0x520639e 0x7662b60 0x128d17e1 0x128cdbbf 0x2b72da6c4ea5 0x2b72da9d7b0d [2025-04-03 14:01:18.488868] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4680902693, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:18.489314] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.492095] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=26][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:19, local_retry_times:19, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:18.492215] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=84][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:18.492233] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=17][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:18.492245] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:18.492308] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=13][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:18.492708] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.494834] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.494883] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:18.499408] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=5) [2025-04-03 14:01:18.501029] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [18288][T1003_Occam][T1003][YB42AC140005-000631D6BF9CDA6D-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-04-03 14:01:18.504677] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CF2-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.508241] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B2A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.510916] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.511031] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.511710] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.511749] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.511769] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.511798] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.511827] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=24][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.521094] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.526754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B2D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.543989] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.544075] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=78] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56396 [2025-04-03 14:01:18.544171] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56396 [2025-04-03 14:01:18.544197] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.549458] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CF6-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.549550] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CF6-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1165, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.560497] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CF7-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.560574] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CF7-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=833, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.566310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.566954] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.568093] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4813482415, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:18.568157] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=57] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.583250] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.587450] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4681001436, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:18.587521] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=49] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.588293] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660078587348]}) [2025-04-03 14:01:18.589108] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660078588342]}) [2025-04-03 14:01:18.590061] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660078589144]}) [2025-04-03 14:01:18.590778] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660078590111]}) [2025-04-03 14:01:18.592025] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660078591244]}) [2025-04-03 14:01:18.592058] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] gts statistics(tenant_id=1004, gts_rpc_cnt=0, get_gts_cache_cnt=306, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-04-03 14:01:18.592256] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=55] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:18.592859] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=47] Wash time detail, (compute_wash_size_time=252, refresh_score_time=110, wash_time=19) [2025-04-03 14:01:18.604936] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.604985] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=42] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.605093] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56398 [2025-04-03 14:01:18.610450] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.611090] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.611128] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.611139] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.612748] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.621162] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.622529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B32-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.623603] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.625364] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72f767e270 fd=129 [2025-04-03 14:01:18.625405] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=31] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56402 [2025-04-03 14:01:18.625495] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56402 [2025-04-03 14:01:18.625513] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.627336] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CFD-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.627387] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CFD-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=764, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.637998] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CFE-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.638059] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CFE-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=509, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.645869] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.645936] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=59] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56404 [2025-04-03 14:01:18.646039] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56404 [2025-04-03 14:01:18.646063] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.646684] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.649480] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CFF-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.649556] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3CFF-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1271, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.659484] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17893][T1001_Occam][T1][YB42AC140005-000631D6BD7CDA1F-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.659694] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:18.661474] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D00-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.661543] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D00-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1691, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.664218] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=4813579015, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:18.664274] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=49] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.664772] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B33-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.667117] INFO [PALF] log_loop_ (log_loop_thread.cpp:175) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=38] LogLoopThread round_cost_time(us)(round_cost_time=62) [2025-04-03 14:01:18.667586] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=32][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.672292] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.674409] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18415][T1003_TenantWea][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.675141] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.675191] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.676338] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.683591] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=41][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.684199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.686877] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.686919] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.690158] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.699529] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=4) [2025-04-03 14:01:18.702957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B34-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.705155] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.705211] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=49] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56406 [2025-04-03 14:01:18.705281] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56406 [2025-04-03 14:01:18.705294] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.706103] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=44] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.708150] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56408 [2025-04-03 14:01:18.711182] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.711231] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.711259] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.711297] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.716994] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D05-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.717120] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D05-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1178, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.719509] INFO [ARCHIVE] get_send_task_ (ob_archive_sender.cpp:364) [18252][T1002_ArcSender][T1002][YB42AC140005-000631D6BE5CB54C-0-0] [lt=874] task not exist, wait time(time_cost=100078) [2025-04-03 14:01:18.721541] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.728534] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D06-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.728615] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D06-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1223, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.739669] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D07-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.739775] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D07-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=925, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.742121] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.742161] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=32] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56410 [2025-04-03 14:01:18.742254] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56410 [2025-04-03 14:01:18.742305] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=46] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.749542] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=33] start do ls ha handler(ls_id_array_=[{id:1}, {id:1002}, {id:1001}, {id:1003}]) [2025-04-03 14:01:18.750723] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D08-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.750800] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D08-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=817, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.750979] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [17796][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=122] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:812, free_obj_hold_cnt:138, allocator used:3777200, allocator total:3916800}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:120, free_obj_hold_cnt:2, allocator used:8002102, allocator total:8002102}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=11918902, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:30, free_obj_hold_cnt:19, allocator used:97216, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=932) [2025-04-03 14:01:18.752736] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.752789] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=45] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56412 [2025-04-03 14:01:18.752885] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56412 [2025-04-03 14:01:18.752902] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.762221] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D09-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.762335] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D09-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1303, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.764438] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.767254] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.767476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.768122] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4813682904, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:18.768170] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.770729] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:18311, tid:18133}]) [2025-04-03 14:01:18.773221] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0A-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.773299] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0A-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=791, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.775144] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [18419][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=13] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-04-03 14:01:18.775238] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18419][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=58] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-04-03 14:01:18.775262] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [18419][T1003_ObTimer][T1003][Y0-0000000000000000-0-0] [lt=19] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=2376) [2025-04-03 14:01:18.782512] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=129 [2025-04-03 14:01:18.782563] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=43] [ussl] accept new connection, fd:129, src_addr:172.20.0.4:56414 [2025-04-03 14:01:18.782652] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:129, src_addr:172.20.0.4:56414 [2025-04-03 14:01:18.782680] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] PNIO dispatch fd to certain group, fd:129, gid:0x100000000 [2025-04-03 14:01:18.783437] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.784418] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.784499] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0B-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1000, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.789091] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4681203074, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:18.789142] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.789579] INFO [CLOG] run1 (ob_garbage_collector.cpp:1352) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0] Garbage Collector is running(seq_=1181, gc_interval=10000000) [2025-04-03 14:01:18.790022] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.790061] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.790076] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.790712] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.790739] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.791763] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.791790] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.792556] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.792578] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.793094] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.793114] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.793351] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.793367] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.793722] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.793738] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.794200] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.794219] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.794701] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.794719] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.795077] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.795094] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.795557] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.795574] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.795771] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0C-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.795876] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0C-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1179, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.795936] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.795943] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.796275] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660078795503]}) [2025-04-03 14:01:18.796401] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.796418] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.796803] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.796838] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.797183] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.797199] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.797675] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.797685] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.798055] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660078796357]}) [2025-04-03 14:01:18.799190] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660078798101]}) [2025-04-03 14:01:18.799422] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.799440] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.799491] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.800333] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.800355] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.800648] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=3] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:18.800735] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.800745] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.800956] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.800982] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.801193] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.801207] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.801318] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=106] Wash time detail, (compute_wash_size_time=307, refresh_score_time=137, wash_time=19) [2025-04-03 14:01:18.801579] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.801592] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.801885] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.801902] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.801983] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.802037] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=52][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802098] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.802120] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802166] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.802187] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=21][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802313] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.802326] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802334] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.802345] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802452] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.802472] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802540] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.802548] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.802560] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802561] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802778] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.802783] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.802929] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.802933] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.803058] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.803072] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.803154] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.803163] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.803234] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1002, wash_size=6291456) [2025-04-03 14:01:18.803242] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.803276] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1445) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0] gc_check_member_list_ cost time(ret=0, time_us=13662) [2025-04-03 14:01:18.803290] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1717) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=9] execute_gc cost time(ret=0, time_us=0) [2025-04-03 14:01:18.803447] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.803471] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.803790] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.803840] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=50][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.804581] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.804614] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=32][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.805405] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.805438] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=32][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.805468] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.805476] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.805652] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.805663] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.805672] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.805682] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.805751] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.805763] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.805999] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.806003] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.806228] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.806241] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.806435] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.806492] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.806498] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.806496] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=60][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.806637] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.806653] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.806684] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.806698] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.806888] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.806897] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.807126] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0D-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.807202] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.807207] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.807202] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0D-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1117, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.807429] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.807442] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.807614] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.807631] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.807751] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.807759] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.807766] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.807781] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.807780] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=6436, clean_start_pos=692054, clean_num=62914) [2025-04-03 14:01:18.808022] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660078799226]}) [2025-04-03 14:01:18.808067] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.808082] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.808155] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.808166] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.808312] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.808320] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.808408] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.808432] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=23][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.808543] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.808551] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.808884] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.808861] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660078808342]}) [2025-04-03 14:01:18.808903] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.809277] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.809312] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.809309] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=31][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.809320] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.809464] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:18.809476] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:18.810324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.811257] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.811320] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=56][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.811358] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.811389] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.811656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.814052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B36-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.816232] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=44] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.818319] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.818426] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0E-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1023, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.819268] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18235][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.821621] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.828992] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D0F-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.829831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B38-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.834264] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.839582] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D10-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.847555] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.848451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.849861] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17560][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.850856] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=1] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.850858] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:18.851490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.851650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.851823] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.852020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.852298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.852449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.852602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.852861] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:18.853215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.853549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.853752] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.854069] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.854573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.855217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.855373] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:18.855885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.856295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.856713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.856970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.857144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.857488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.858138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.858657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.858973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.859141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.859329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.859670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.859938] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6D8-0-0] [lt=2190][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.860193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.860838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.861178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.861397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.861398] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D12-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.862150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.862170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.862531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.862545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.862835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.862843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.863033] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.863042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.863282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.863465] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC89F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.863583] INFO [CLOG] get_ls_status_from_table (ob_garbage_collector.cpp:1384) [18207][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0] failed to get ls status info from table(ret=-4013, tenant_id=1002, ls_id={id:1001}) [2025-04-03 14:01:18.864677] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=4813778863, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:18.864701] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.865645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.865659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.865833] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.865842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.865979] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.865986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.866455] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.866465] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.866780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.866988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.867108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.867284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.867318] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.867445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.867608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.867793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.868179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.868316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.868415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.868732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.869243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.869252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.869415] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.869597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.869837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.870192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.870204] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.870523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.870683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.871198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.871209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.871358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.871566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.871671] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.871873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.872026] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=152] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.872335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.872558] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.872836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.873088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.873331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.873236] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D13-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.873482] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D13-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1879, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.873712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.873989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.874157] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.874323] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.874467] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.874610] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.874775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.874963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.875105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.875300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.875443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.875601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.875739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.875897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.876273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.876390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.876541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.877135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.877237] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.877375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.877621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.877711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.877894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.878082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.878426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.878581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.878678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.878878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.879164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.879419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.879645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18207][T1002_GCCollect][T1001][YB42AC140005-000631D6C43CC8A0-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.883498] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.884316] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D14-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.884372] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D14-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=707, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.889247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889717] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.889988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890175] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890544] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.890899] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4681305444, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:18.890917] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.895064] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D15-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.895127] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D15-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=606, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.899663] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=45] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=3) [2025-04-03 14:01:18.905700] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D16-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.905753] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D16-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=465, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.910874] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660078909430]}) [2025-04-03 14:01:18.911910] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660078910969]}) [2025-04-03 14:01:18.912122] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.912162] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.912181] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.912203] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] re_register_parent reason(palf_id=1003, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660074907478, last_first_register_time_us=1743648247119503, register_time_us=1743660074907478, curr_time_us=1743660078912201) [2025-04-03 14:01:18.912301] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912432] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912746] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660078911939]}) [2025-04-03 14:01:18.912761] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.912995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913423] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660078912772]}) [2025-04-03 14:01:18.913560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913693] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.913911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914173] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.914912] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660078913864]}) [2025-04-03 14:01:18.914966] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1003, self="172.20.0.5:2882") [2025-04-03 14:01:18.914994] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.916279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.916305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.916520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.916551] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.916933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.916946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.917339] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.917352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.917335] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D17-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.917400] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D17-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1475, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.917512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.917533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.917703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.917716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.917913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.917926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918409] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.918839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.919306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.919659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.919952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.920099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.920247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.920692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.920859] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.921197] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.921222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.921607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.921720] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:120) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=33] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"172.20.0.5:2882", switch_to_leader_ts:-1}) [2025-04-03 14:01:18.921763] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:18.922144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.922192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=46] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.922863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.922883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.923308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.923324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.923631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.923769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.924183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.924264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.924435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.924592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.924770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.924936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.925285] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.925442] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.925729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.926063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.926224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.926368] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.926901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.927069] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.927280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.927363] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.927700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.928163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.928595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.928681] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D18-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.929278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.929740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.930036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.930396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.930862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.930956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.931120] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.931270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933241] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933342] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933406] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.933866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.934042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.934367] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.934515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.934706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.935040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.935202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.935528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.935854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.935994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.936153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.936799] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.936995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.937142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.937325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.937461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.937828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.938271] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.938420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.938765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.938919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.939432] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.949871] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1A-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.949912] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1A-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=319, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.960415] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.960449] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1B-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=390, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.964745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.964760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.965167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.965181] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.965418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.965429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.965655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.965666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.965901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.965913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.966106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.966116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.966388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.966407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.966602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.966613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.967037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.967053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.967358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.967372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.967401] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=35][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.967780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.967800] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.968028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.968054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.968288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.968306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.968525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.968538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.968868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.968880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.969210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.969223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:18.969345] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.969599] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4813883944, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:18.969642] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.970031] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=41] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}, {id:1002}]) [2025-04-03 14:01:18.971160] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=36][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4555, tid:17463}]) [2025-04-03 14:01:18.971420] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:18.971440] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=18][errcode=-4013] oops, alloc failed, tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=9223372036854775807, tenant_hold=8096894976, tenant_limit=9223372036854775807 [2025-04-03 14:01:18.971453] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=9][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:18.971462] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=8][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:18.971484] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=21][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:18.971491] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=6][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:18.971500] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=5][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:18.971535] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=34] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.971556] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=20][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.971596] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=39] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=935, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.971616] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1C-0-0] [lt=19][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.982235] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=19][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:18.982271] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=34][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:18.982282] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=10][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:18.982290] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=7][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:18.982302] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=8][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:18.982386] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=82] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.982420] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=33][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.982464] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=43] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=608, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.982496] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1D-0-0] [lt=31][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.983569] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:18.989310] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:831) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-04-03 14:01:18.989370] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=4681403345, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:18.989388] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:18.994109] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=47][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:18.994164] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=54][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:18.994179] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=14][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:18.994192] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=11][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:18.994208] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=9][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:18.994281] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=71] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:18.994332] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=43][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.994449] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=115] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1685, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:18.994487] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1E-0-0] [lt=36][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.005439] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=35][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.005484] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=43][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:19.005518] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=33][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:19.005585] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=60][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.005624] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=31][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:19.005723] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=97] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.005767] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=42][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.005856] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=87] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1179, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.005926] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D1F-0-0] [lt=68][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.008419] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=47] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:19.008726] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash time detail, (compute_wash_size_time=158, refresh_score_time=63, wash_time=11) [2025-04-03 14:01:19.009866] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [17844][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=35] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:19.012191] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=47][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.012215] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.012238] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.012248] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.012300] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=3555, clean_start_pos=754968, clean_num=62914) [2025-04-03 14:01:19.017092] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=59][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.017124] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=31][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:19.017135] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=10][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:19.017144] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=8][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.017157] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=7][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:19.017216] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=58] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.017275] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=58][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.017319] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=43] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1174, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.017342] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D20-0-0] [lt=22][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.018680] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.018706] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.018717] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660079018034], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.018731] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660079018034], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.018761] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079018034]}) [2025-04-03 14:01:19.020625] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.020639] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.020647] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660079018779], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.020656] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660079018779], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.020684] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079018779]}) [2025-04-03 14:01:19.021359] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.021374] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.021383] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660079020694], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.021390] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660079020694], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.021404] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660079020694]}) [2025-04-03 14:01:19.021843] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:134) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=15] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1004, epoch:-1, self:"172.20.0.5:2882", switch_to_leader_ts:-1}) [2025-04-03 14:01:19.021888] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.022957] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.022972] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.022979] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660079021411], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.023002] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660079021411], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.023031] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660079021411]}) [2025-04-03 14:01:19.024538] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.024557] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.024564] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660079023245], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.024573] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660079023245], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.024587] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660079023245]}) [2025-04-03 14:01:19.026547] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.026590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.026620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.026968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.026993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.027230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.027251] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.027554] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.027578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.027712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.027733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.027962] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.027981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.028476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.028501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.028533] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=22][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.028565] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=31][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:19.028579] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=13][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:19.028595] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=15][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.028616] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=12][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:19.028660] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=43] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.028692] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=31][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.028718] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=25] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1240, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.028757] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D21-0-0] [lt=38][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.029386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.029411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.030042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.030063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.030591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.030615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.031308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.031365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=55] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.032085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.032105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.033055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.033112] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=54] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.033793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.033886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=91] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034708] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034725] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034881] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034979] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.034994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.035965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036534] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036625] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036632] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.036665] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.036676] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=10][errcode=-4013] rpc memory pool alloc memory failed(sz=1340, alloc_sz=16256) [2025-04-03 14:01:19.036686] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9][errcode=-4013] alloc buffer fail(payload_sz=1156) [2025-04-03 14:01:19.036870] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=182][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.036884] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=7][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=989669, timeout_timestamp=1743660080021975) [2025-04-03 14:01:19.036897] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=11][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:19.036908] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=7][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:19.036925] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:19.036934] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=7][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:19.036942] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=5][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:19.036949] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=6][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:19.036962] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=11][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:19.036987] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=21][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:19.037047] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=47][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:19.037055] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=8][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:19.037062] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=7][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:19.037333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.037350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.037546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.037561] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.037746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.037761] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.037963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.037997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038424] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.038751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039508] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=41][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.039542] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=33][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:19.039547] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039561] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=18][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:19.039563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039571] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=9][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.039584] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=8][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:19.039630] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=44] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.039678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039668] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=24][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.039702] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=33] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=768, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.039735] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D22-0-0] [lt=32][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.039770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.039880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040470] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040641] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.040932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041616] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.041973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042212] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042579] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.042915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043242] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043416] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.043956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044254] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044616] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044819] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044922] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.044931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.045140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.045151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.045269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.045280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.045788] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.045805] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.045978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.045997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.046968] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:19.046978] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=0][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:19.046989] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:19.046997] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=6][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:19.047019] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=21][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:19.047028] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:19.047035] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=6][errcode=-4013] query failed(ret=-4013, conn=0x2b739baea050, start=1743660079021991, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:19.047047] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:19.047123] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=16][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:19.047134] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=9][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1002, ls_ids=[{id:1}]) [2025-04-03 14:01:19.047151] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=11][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1002, ls_ids=[{id:1}]) [2025-04-03 14:01:19.047164] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=11][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1002, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660079021396}) [2025-04-03 14:01:19.047173] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=8][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:19.047185] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3E-0-0] [lt=11][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:19.048113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048128] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048235] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048486] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048891] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.048963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049540] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049911] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.049923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050066] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050313] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050387] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050567] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050655] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050664] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.050997] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.051020] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=1374, alloc_sz=16256) [2025-04-03 14:01:19.051027] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1190) [2025-04-03 14:01:19.051045] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=17][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.051053] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1003, svr="172.20.0.3:2882", timeout=998043, timeout_timestamp=1743660080047290) [2025-04-03 14:01:19.051065] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=11][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:19.051145] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=4][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:19.051131] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=27][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.051155] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:19.051161] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=28][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:19.051162] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:19.051176] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=13][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:19.051176] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=14][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:19.051187] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=10][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:19.051190] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=13][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.051196] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8][errcode=-4013] execute failed(ret=-4013, tenant_id=1003, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:19.051205] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:19.051226] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=14][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:19.051231] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:19.051235] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=4][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:19.051211] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=14][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:19.051276] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=63] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.051314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051310] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=33][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.051343] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=32] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1387, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.051375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051388] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D23-0-0] [lt=43][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.051436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051442] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051960] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.051972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052342] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052403] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052708] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.052973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053038] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053409] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053542] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053671] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053969] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.053977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054126] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054219] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054380] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054387] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054629] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054734] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054946] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.054956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055060] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055112] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055219] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055539] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.055791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056575] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.056642] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:19.056650] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:19.056658] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:19.056664] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:19.056670] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1003) [2025-04-03 14:01:19.056680] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=10][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1003) [2025-04-03 14:01:19.056686] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=5][errcode=-4013] query failed(ret=-4013, conn=0x2b73ba17a050, start=1743660079047296, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:19.056694] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:19.056737] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004) [2025-04-03 14:01:19.056743] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:19.056751] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=7][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:19.056758] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1004, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660079024580}) [2025-04-03 14:01:19.056765] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=6][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:19.056769] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B3F-0-0] [lt=3][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:19.057740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.057763] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.057866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.057876] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.057955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.057965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058140] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058298] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058384] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.058991] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.059142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.059338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.059452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.059604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.059793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.059941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.060125] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.060137] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=11][errcode=-4013] rpc memory pool alloc memory failed(sz=1340, alloc_sz=16256) [2025-04-03 14:01:19.060142] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=4][errcode=-4013] alloc buffer fail(payload_sz=1156) [2025-04-03 14:01:19.060147] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.060155] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=998064, timeout_timestamp=1743660080056798) [2025-04-03 14:01:19.060168] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=13][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:19.060176] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=7][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:19.060181] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:19.060185] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=3][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:19.060235] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.060617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.060757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.060893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.061392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.061404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.061764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.062031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.062200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.062353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.062528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.062844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.063024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.063518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.063696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.063818] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.064158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.064330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.064537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.064974] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B40-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.066827] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.066841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.066906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.066914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.067170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.067462] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.067642] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.067768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.067890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.068293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.068393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.068556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.068723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.068889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.069358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.069517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.070046] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.070675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.070846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.070989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.071166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.071203] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=44][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.071352] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.071416] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:8977, tid:17422}]) [2025-04-03 14:01:19.071457] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071510] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071550] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071595] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071646] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071653] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.071644] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.071692] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071730] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071770] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071824] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071907] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071956] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.071970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.072002] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072051] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072040] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 6 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.072095] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=6361][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.072113] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072151] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072194] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072236] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072283] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.072340] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072382] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072422] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.072463] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B41-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:19.073144] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.073185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.073341] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.073406] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.073520] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.073536] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.073625] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.073714] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.073773] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.073780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.073853] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.073892] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074108] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.074164] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074265] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074326] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074386] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074392] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=5] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.074496] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074550] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074655] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.074708] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074695] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.074746] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074797] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074858] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.074961] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075023] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075082] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075131] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075185] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075193] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.075243] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.075305] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075355] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075400] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075502] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075596] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075665] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075721] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075766] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075829] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.075883] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075936] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.075941] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.075982] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076041] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076102] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=8] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.076174] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076223] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076252] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.076270] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076319] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076383] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076403] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.076506] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076565] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076615] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076652] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.076665] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.076719] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076769] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076830] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076879] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076932] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076981] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.076976] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.077041] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077093] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077142] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077151] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.077195] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077248] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077292] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077345] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077394] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077448] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077454] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.077442] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.077500] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077552] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077598] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077647] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077692] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077705] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.077796] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077792] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.077855] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077904] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.077959] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078022] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078068] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078117] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078162] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.078220] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078204] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.078266] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078316] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078369] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078416] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078444] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.078467] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078468] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.078513] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078566] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078615] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078671] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.078703] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078750] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078804] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078862] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.078873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.078864] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.078917] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.079018] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.079069] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.079143] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.079154] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.079217] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.079267] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.079321] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B42-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.079470] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.079501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.079662] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.079803] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.079846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=41] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.079969] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.080279] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.080300] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.080418] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.080548] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 2249 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 194 | get_cnt= 312 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 23 | get_cnt= 32 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 55 | get_cnt= 94 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 310 | get_cnt= 784 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 259 | get_cnt= 67322 | score=4.911944 | [CACHE] tenant_id= 1 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 18 | get_cnt= 3982 | score=0.928150 | [CACHE] tenant_id= 1 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 10 | get_cnt= 8644 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 14 | get_cnt= 435 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 2530 | get_cnt= 9318 | score=824.358623 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 4 | status= 1 | policy= 1 | kv_cnt= 8 | get_cnt= 646811 | score=127184.785829 | [CACHE] tenant_id= 1 | cache_id= 0 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 186 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 69 | get_cnt= 69 | score=0.000000 | [CACHE] tenant_id= 1 | cache_id= 3 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 1 | get_cnt= 1915 | score=957.407386 | [2025-04-03 14:01:19.083699] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.088212] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D26-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.088262] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D26-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=782, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.089423] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089496] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089569] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089663] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089675] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089736] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089750] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089780] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089794] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089835] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089882] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089905] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089941] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.089952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.089990] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.090000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090040] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.090054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090084] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.090094] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090121] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.090130] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090163] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.090177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090181] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090222] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.090283] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.090297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.090341] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.090389] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:19.098937] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D27-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.098997] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D27-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=506, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.099793] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=2) [2025-04-03 14:01:19.103974] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:19.104058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=78] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.104079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.104097] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=16][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.105364] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:19.105407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.105435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.105767] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:19.105797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=28] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.105825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.106161] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:19.106199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.106211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.107576] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 45 | get_cnt= 64 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 125 | get_cnt= 284 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 4 | get_cnt= 370 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 135 | get_cnt= 30636 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 64 | get_cnt= 369 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 32 | get_cnt= 105 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 202 | get_cnt= 202 | score=0.000000 | [CACHE] tenant_id= 1002 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 5 | get_cnt= 5 | score=0.000000 | [2025-04-03 14:01:19.107613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.107628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.108349] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.108381] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.109024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.109055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.109621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.109647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.109932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.109894] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D28-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.109953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.109959] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D28-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=766, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.110144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110168] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110293] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110481] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110662] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110686] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110922] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.110944] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.111187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.111209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.111385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.111404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.111560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.111580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.111728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.111748] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.112294] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.112353] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=52][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.112374] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.112396] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.115059] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115324] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115589] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.115971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116219] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.116892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117152] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117559] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117570] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117757] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.117967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.117987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118195] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118599] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118782] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.118983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.119131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.119448] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.119745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.120158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.120789] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D29-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.120894] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D29-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=741, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.121262] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.121929] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.122308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.122768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.122781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.124326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.124341] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.124583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.125165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.125453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.125649] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.125765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.126017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.126094] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.126308] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.126448] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079125877]}) [2025-04-03 14:01:19.126578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.126762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.126923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.127071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.127259] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.127562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.127724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.127886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.128073] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.128221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DA-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.128701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.128749] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:261) [18244][T1002_ArcSender][T1002][YB42AC140005-000631D6BE1CB54C-0-0] [lt=2725][errcode=-4018] try free send task failed(ret=-4018) [2025-04-03 14:01:19.128868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.129204] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.129365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.129531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.129696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.130068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.130247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.130678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.131228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.131316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.131511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.131648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.131839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.131961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.132305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.132501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.132606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.132935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.133470] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.133767] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.134129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.134247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.134598] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.134742] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B43-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.135943] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.135955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.136014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.136248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.136511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.136690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.137217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.137513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.137527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.137686] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.137831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.137998] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.138182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.138310] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.138660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.138953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.139457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.139796] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.139940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.140126] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.140272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.140421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.140914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.141286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.141414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.141579] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.141909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.142080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.142227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.142704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B44-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.143471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B45-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.143523] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B45-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.143688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B45-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.143845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B45-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.155841] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D2C-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.155907] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D2C-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1811, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.156967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B47-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.157068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B47-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.157143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B47-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.157216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B47-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.158176] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B47-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.158177] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=92] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:19.158494] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154D-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.163118] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B47-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.166688] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D2D-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.167536] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.168295] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:19.169555] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.171199] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154D-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.172515] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.180200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18246][T1002_ArcSrv][T1001][YB42AC140005-000631D6BE3CB54C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.183756] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=62][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.194479] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.194525] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28648 [2025-04-03 14:01:19.194603] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28648 [2025-04-03 14:01:19.194779] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=171] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.195960] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [18303][T1003_PalfGC][T1003][Y0-0000000000000000-0-0] [lt=63] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):1024, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):1024, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):255, sequence:1}) [2025-04-03 14:01:19.198396] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D30-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.198428] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D30-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=392, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.198723] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.198761] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=31] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28650 [2025-04-03 14:01:19.198885] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28650 [2025-04-03 14:01:19.198913] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.200847] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [17813][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=59] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-04-03 14:01:19.202176] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.202234] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=49] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28652 [2025-04-03 14:01:19.202309] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28652 [2025-04-03 14:01:19.202323] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.207281] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.207435] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=145] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28654 [2025-04-03 14:01:19.207514] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28654 [2025-04-03 14:01:19.207542] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.209031] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D31-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.209074] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D31-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=540, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.212374] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.212409] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29] re_register_parent reason(palf_id=1002, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660075207726, last_first_register_time_us=1743648247019455, register_time_us=1743660075207726, curr_time_us=1743660079212408) [2025-04-03 14:01:19.212475] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.212953] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.213015] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.213066] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.213044] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=14] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:19.213131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.213200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.213283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.213333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.213347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.213386] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1002, self="172.20.0.5:2882") [2025-04-03 14:01:19.213402] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.213417] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.213429] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.213670] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=40] Wash time detail, (compute_wash_size_time=233, refresh_score_time=131, wash_time=19) [2025-04-03 14:01:19.219715] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D32-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.219743] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D32-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=600, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.220953] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=7243, clean_start_pos=817882, clean_num=62914) [2025-04-03 14:01:19.222021] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.229030] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17722][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=39] ====== tenant freeze timer task ====== [2025-04-03 14:01:19.230227] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079229493]}) [2025-04-03 14:01:19.230349] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D33-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.230376] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D33-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=532, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.230771] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079230276]}) [2025-04-03 14:01:19.231658] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660079230801]}) [2025-04-03 14:01:19.232101] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660079231685]}) [2025-04-03 14:01:19.233389] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660079232471]}) [2025-04-03 14:01:19.233909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.234218] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.234264] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=26] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:19.237488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.238838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.240357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.240380] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.240916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.241081] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D34-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.241481] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.241782] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.245793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.248446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.249735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B49-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.249856] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FE-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.249881] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.252469] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.252991] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D35-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.253142] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D35-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1816, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.253772] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FE-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.257528] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FE-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.258015] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B4A-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.258495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B4A-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.259510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B4A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.263733] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.263781] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=41] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28658 [2025-04-03 14:01:19.263883] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28658 [2025-04-03 14:01:19.263882] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D36-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.263901] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.267634] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.272201] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=217] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.272263] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=54] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28660 [2025-04-03 14:01:19.272336] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.274870] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D37-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.274952] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D37-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=830, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.275843] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.275870] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28662 [2025-04-03 14:01:19.275925] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28662 [2025-04-03 14:01:19.275961] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=33] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.277577] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=0] ====== tenant freeze timer task ====== [2025-04-03 14:01:19.277618] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1555) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29] [TenantFreezer] tenant have inactive memstores(ctx.freezable_active_memstore_used_=25165824, ctx.total_memstore_used_=58720256, ctx.total_memstore_hold_=58720256, memstore_freeze_trigger_limit_=43837840, tenant_id=1001) [2025-04-03 14:01:19.277643] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1564) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18] [TenantFreezer] oldest frozen memtable(list="") [2025-04-03 14:01:19.283706] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [18408][T1003_LCLSender][T1003][Y0-0000000000000000-0-0] [lt=992] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b734e9e82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-04-03 14:01:19.283929] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.285959] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D38-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.286083] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D38-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=928, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.289663] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.289727] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290377] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290582] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290686] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290791] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.290992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.291135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.291275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.291386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.291487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.291592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.291695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.291801] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4681705661, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:19.291883] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=75] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:19.294260] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE379-0-0] [lt=7673][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.295179] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.295202] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28664 [2025-04-03 14:01:19.295249] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28664 [2025-04-03 14:01:19.295280] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=28] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.297315] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D39-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.297405] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D39-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1076, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.299112] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.299149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.299898] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=53] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=1) [2025-04-03 14:01:19.302123] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.302341] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.302531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.302713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.302954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.303208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.303466] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.303658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.303863] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.304066] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.304269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.304447] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.304623] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.304844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.305239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.305463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.305599] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.305668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.305855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.306033] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.306146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.306231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.306366] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.306449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.306522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.306596] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.306677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.310244] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.311767] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.311954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E4-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.312483] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.312529] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.312549] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.312567] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.320088] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D3B-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.322080] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.327243] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.327279] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28666 [2025-04-03 14:01:19.327343] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28666 [2025-04-03 14:01:19.327366] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.331205] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D3C-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.331285] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D3C-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=929, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.334598] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079334058]}) [2025-04-03 14:01:19.337050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B4C-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.346666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B4D-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.346893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B4D-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.351312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B4D-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.353315] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D3E-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.361546] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=22][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:19.361590] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=41][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660105653097, now=[mts=1743660078140987], now0=[mts=1743660078140987], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:19.361625] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=31][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a431d2a0, tx_id:{txid:76175014}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660078133205, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:19.361775] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=141][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b739ba5c1a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a431d2a0}) [2025-04-03 14:01:19.361844] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=67][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a431d2a0, tx_id:{txid:76175014}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660078133205, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:19.361903] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=56] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f8054008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660078138611, tx_desc={this:0x2b73a431d2a0, tx_id:{txid:76175014}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660078133205, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660075723113, use_das=false, nested_level=0, session={this:0x2b739ba5c1a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a431d2a0}, plan=0x2b7338b48050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:19.362030] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=125][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:19.362081] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=48][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:19.362101] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=18][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:19.362112] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=9][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:19.362145] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=31][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:19.362166] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=18][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=2, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:19.362204] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=34] will sleep(sleep_us=2000, remain_us=26360894, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=2, v.err_=-4283, timeout_timestamp=1743660105723097) [2025-04-03 14:01:19.364313] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=26][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:2, local_retry_times:2, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:19.364380] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=46][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:19.364402] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=20][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:19.364412] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=9][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:19.364509] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.364560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.364706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.364719] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D3F-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.364839] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.364804] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D3F-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1260, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.364928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.365029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.367715] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.370290] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.370335] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:19.372051] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:19.376230] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D40-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.376311] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D40-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1184, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.379423] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.379497] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=64] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28670 [2025-04-03 14:01:19.379584] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28670 [2025-04-03 14:01:19.379609] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.383990] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.386909] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D41-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.386946] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D41-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=347, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.389719] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=206] get wrs ts(ls_id={id:1}, delta=4681803172, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:19.389774] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=47] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:19.395347] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.395392] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=36] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28672 [2025-04-03 14:01:19.395477] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28672 [2025-04-03 14:01:19.395505] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.397615] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D42-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.397648] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D42-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=523, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.407715] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.407765] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=42] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28674 [2025-04-03 14:01:19.407897] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28674 [2025-04-03 14:01:19.407920] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.408560] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D43-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.408599] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D43-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=836, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.412576] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.412631] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.412653] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.412671] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.412770] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=1] ====== traversal_flush timer task ====== [2025-04-03 14:01:19.412980] INFO [MDS] flush (mds_table_mgr.cpp:172) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=20] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b73c068a510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616808737813004, v:0}, clog_base_lsn:{lsn:4697333760}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743622739127262315, v:0}, all_id_meta:{id_meta:[{limited_id:1743655270710910399, latest_log_ts:{val:1743655254062460505, v:0}}, {limited_id:35000001, latest_log_ts:{val:1743639632198700904, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:450, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:19.413276] INFO [MDS] flush (mds_table_mgr.cpp:172) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=187] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b73b7ede510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1743655263788752080, v:0}, clog_base_lsn:{lsn:3690762240}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743648258823515129, v:0}, all_id_meta:{id_meta:[{limited_id:1740569710308384976, latest_log_ts:{val:1740569693675557106, v:0}}, {limited_id:32000001, latest_log_ts:{val:1740551265169597432, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:1292, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:19.413469] INFO [MDS] flush (mds_table_mgr.cpp:172) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=128] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b73b7f94510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1002, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1743655484668882629, v:0}, clog_base_lsn:{lsn:3556552704}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659970732747227, v:0}, tablet_change_checkpoint_scn:{val:1743648258592483359, v:0}, all_id_meta:{id_meta:[{limited_id:1743494462089433682, latest_log_ts:{val:1743494445468388405, v:0}}, {limited_id:33000001, latest_log_ts:{val:1743494442089433683, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:171, palf_env_:0x2b7332bfc030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:19.413571] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [18237][T1002_Flush][T1002][Y0-0000000000000000-0-0] [lt=97] succeed to traversal_flush(ret=0, ls_cnt=3) [2025-04-03 14:01:19.419153] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D44-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.419181] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D44-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=457, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.421707] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=92] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:19.422169] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.422448] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=55] Wash time detail, (compute_wash_size_time=229, refresh_score_time=134, wash_time=31) [2025-04-03 14:01:19.429672] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D45-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.429696] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D45-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=419, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.429739] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=7257, clean_start_pos=880796, clean_num=62914) [2025-04-03 14:01:19.436040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B4E-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.436273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B4E-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.436452] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B4E-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.438775] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.438889] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079437597]}) [2025-04-03 14:01:19.439717] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079438933]}) [2025-04-03 14:01:19.440193] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660079439754]}) [2025-04-03 14:01:19.440257] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=62] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660079439754]}) [2025-04-03 14:01:19.440359] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D46-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.440391] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D46-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=598, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.440971] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660079440280]}) [2025-04-03 14:01:19.441721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B4E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.441896] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660079441376]}) [2025-04-03 14:01:19.451169] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D47-0-0] [lt=0][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.451192] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D47-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.455901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B4F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.461711] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D48-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.461741] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D48-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=444, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.467806] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=39][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.468206] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B50-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.472328] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.472985] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D49-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.473107] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D49-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1202, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.475313] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.478397] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.484074] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.490758] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4681904970, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:19.490785] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:19.495076] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=140 [2025-04-03 14:01:19.495096] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:140, src_addr:172.20.0.3:28684 [2025-04-03 14:01:19.495145] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28684 [2025-04-03 14:01:19.495156] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] PNIO dispatch fd to certain group, fd:140, gid:0x100000000 [2025-04-03 14:01:19.496731] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4B-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.496838] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4B-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1894, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.500085] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=44] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=0) [2025-04-03 14:01:19.504522] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:19.504551] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28686 [2025-04-03 14:01:19.504606] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28686 [2025-04-03 14:01:19.504617] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.507580] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4C-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.507638] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4C-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=639, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.513113] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.514686] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.514765] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.514827] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.515940] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.515978] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.518579] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4D-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.522279] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.526430] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.526563] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28688 [2025-04-03 14:01:19.528369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B51-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.529900] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.530942] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4E-0-0] [lt=2] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2154, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.542142] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4F-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.542236] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D4F-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=990, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.543281] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.543366] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079542633]}) [2025-04-03 14:01:19.543416] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:171) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", gts_tenant_info={v:1}) [2025-04-03 14:01:19.544121] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079543441]}) [2025-04-03 14:01:19.545244] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660079544153]}) [2025-04-03 14:01:19.546224] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660079545475]}) [2025-04-03 14:01:19.546218] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=1] ====== tenant freeze timer task ====== [2025-04-03 14:01:19.546608] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=55][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.547840] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660079546625]}) [2025-04-03 14:01:19.548620] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.548691] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:552) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=49, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=55, Total TxDataTable Memory Limit(MB)=204) [2025-04-03 14:01:19.560617] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=48] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:19.565301] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D51-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.567861] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=64][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.571892] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.571912] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28690 [2025-04-03 14:01:19.575976] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D52-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.576048] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D52-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=541, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.579419] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:19.579438] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28692 [2025-04-03 14:01:19.579470] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28692 [2025-04-03 14:01:19.579479] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.584140] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=52][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.584387] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.584400] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28694 [2025-04-03 14:01:19.584438] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28694 [2025-04-03 14:01:19.584446] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.586575] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D53-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.586634] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D53-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=448, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.589840] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=4682003917, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:19.589865] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:19.594885] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.594900] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28696 [2025-04-03 14:01:19.594936] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28696 [2025-04-03 14:01:19.594943] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.597465] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D54-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.597562] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D54-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=765, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.600361] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.600389] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28698 [2025-04-03 14:01:19.600434] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28698 [2025-04-03 14:01:19.600443] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.607057] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.607076] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28700 [2025-04-03 14:01:19.607121] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28700 [2025-04-03 14:01:19.607130] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.608651] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D55-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.608743] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D55-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=987, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.613029] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.613052] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.613063] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.613072] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.619597] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D56-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.619676] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D56-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=783, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.622383] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.630282] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:19.630419] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18111][T1002_LogUpdate][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.630734] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=38] Wash time detail, (compute_wash_size_time=181, refresh_score_time=86, wash_time=27) [2025-04-03 14:01:19.630997] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.631034] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=32] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28702 [2025-04-03 14:01:19.631100] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28702 [2025-04-03 14:01:19.631110] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.632029] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.632089] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D57-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.632167] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D57-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2247, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.636269] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5517, clean_start_pos=943710, clean_num=62914) [2025-04-03 14:01:19.639692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B54-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.643218] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D58-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.647358] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=39] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.649386] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079648606]}) [2025-04-03 14:01:19.650203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B55-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.660735] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.660787] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28704 [2025-04-03 14:01:19.667951] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.675632] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D5B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.682506] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.684193] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.686115] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.688858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B5A-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.691220] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.695790] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:19.695829] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=34] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28714 [2025-04-03 14:01:19.695870] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28714 [2025-04-03 14:01:19.698246] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.700805] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=195][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:19.700871] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=64][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660078492604], now0=[mts=1743660078492604], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:19.700914] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=40][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719471}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660078491287, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:19.701026] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=85][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:19.701090] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=62][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719471}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660078491287, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:19.701142] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=48] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.701243] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=99][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:19.701266] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:19.701277] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=9][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:19.701288] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:19.701298] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:19.701331] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=30][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=20, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:19.701352] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=17] will sleep(sleep_us=20000, remain_us=4338109, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=20, v.err_=-4283, timeout_timestamp=1743660084039460) [2025-04-03 14:01:19.702771] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:686) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=60] Cache replace map node details(ret=0, replace_node_count=0, replace_time=2334, replace_start_pos=943710, replace_num=31457) [2025-04-03 14:01:19.702823] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=49] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=10) [2025-04-03 14:01:19.706257] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:19.707718] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D5E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.707753] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D5E-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=267, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.713093] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.713139] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=40][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.713181] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.713198] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.718134] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D5F-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.718187] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D5F-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=364, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.718421] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=27] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.718468] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=36] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=315) [2025-04-03 14:01:19.721454] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=20][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:20, local_retry_times:20, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:19.721513] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=41][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:19.721524] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=10][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:19.721530] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=6][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:19.721547] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:19.722450] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=46][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.722711] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.722739] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:19.726594] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.726642] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28718 [2025-04-03 14:01:19.726725] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28718 [2025-04-03 14:01:19.726748] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.728913] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D60-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.728971] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D60-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=637, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.739912] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D61-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.740027] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D61-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=845, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.749830] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=51] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1002}, {id:1003}, {id:1}]) [2025-04-03 14:01:19.750875] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D62-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.750961] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D62-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=741, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.754458] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079753629]}) [2025-04-03 14:01:19.755191] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079754515]}) [2025-04-03 14:01:19.755231] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=37] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=150, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-04-03 14:01:19.755250] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:516) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] refresh gts(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079754515]}) [2025-04-03 14:01:19.757350] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660079755272]}) [2025-04-03 14:01:19.759717] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1] ====== tenant freeze timer task ====== [2025-04-03 14:01:19.760081] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=38][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.760653] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660079757390]}) [2025-04-03 14:01:19.763649] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D63-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.765484] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.767788] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.768039] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=97][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.768252] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:657) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=1) [2025-04-03 14:01:19.784282] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.785485] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28722 [2025-04-03 14:01:19.785795] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D65-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.790680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B5F-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.795455] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.795494] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28724 [2025-04-03 14:01:19.795591] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28724 [2025-04-03 14:01:19.795607] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.796238] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D66-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.796268] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D66-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=340, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.802865] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=36] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:19.802897] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28726 [2025-04-03 14:01:19.802993] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28726 [2025-04-03 14:01:19.803056] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=59] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.803794] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:116) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-04-03 14:01:19.803846] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=39] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-04-03 14:01:19.803863] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=14] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-04-03 14:01:19.803871] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=7] [tabletchange] task check ls(ls->get_ls_id()={id:1002}, tablet_persist_trigger=0) [2025-04-03 14:01:19.803877] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:150) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=5] [tabletchange] task check ls(ls->get_ls_id()={id:1003}, tablet_persist_trigger=0) [2025-04-03 14:01:19.803885] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:240) [18623][T1004_ObTimer][T1004][Y0-0000000000000000-0-0] [lt=6] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=4, times=2369) [2025-04-03 14:01:19.806719] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D67-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.806752] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D67-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=378, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.813196] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.813252] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.813271] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.813305] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.817247] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D68-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.817296] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D68-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=421, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.822520] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.823846] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=48] PNIO [ratelimit] time: 1743660079823844, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-04-03 14:01:19.827803] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D69-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.827868] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D69-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=432, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.835626] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.835675] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=41] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28728 [2025-04-03 14:01:19.835759] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28728 [2025-04-03 14:01:19.835785] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.836983] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:19.837612] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=95] Wash time detail, (compute_wash_size_time=288, refresh_score_time=110, wash_time=18) [2025-04-03 14:01:19.839033] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6A-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.839115] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6A-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1028, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.843340] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.844983] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=47] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=7321, clean_start_pos=1006624, clean_num=62914) [2025-04-03 14:01:19.846922] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [17564][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=2] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:19.849500] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17560][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:19.850752] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:19.850836] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:19.851890] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.851967] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6B-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=994, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.852982] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:19.858767] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:19.858792] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28730 [2025-04-03 14:01:19.858841] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28730 [2025-04-03 14:01:19.858848] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:19.862968] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6C-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.863090] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6C-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=884, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.865464] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4814779915, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:19.865501] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:19.865670] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079864750]}) [2025-04-03 14:01:19.866233] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079865720]}) [2025-04-03 14:01:19.866746] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17512][SysLocAsyncUp0][T0][YB42AC140005-000631D6BADDA6E5-0-0] [lt=598][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.867666] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660079866263]}) [2025-04-03 14:01:19.868445] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=31][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.869130] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869163] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869401] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=43] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869425] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660079867701]}) [2025-04-03 14:01:19.869518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.869853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.870050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.870084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.870509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.870538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.871603] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660079869847]}) [2025-04-03 14:01:19.871631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.871653] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.871945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.871970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.872383] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.872413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.873085] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:18004, tid:17463}]) [2025-04-03 14:01:19.873141] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.873155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.873167] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.873208] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=40][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.873567] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.873588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.873600] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.873608] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.873802] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6D-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.873878] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6D-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=630, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.874133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.874155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.874179] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=23][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.874190] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.874546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.874566] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.874577] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.874586] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.874740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.874760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.874772] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.874781] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.874972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.875014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.875030] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=15][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.875040] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.875332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.875367] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.875380] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.875388] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.875923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.875939] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.875952] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.875961] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.876193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.876220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.876234] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.876266] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=31][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.876619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.876643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.876656] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.876664] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.876804] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.876844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.876857] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.876874] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=17][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.877209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.877226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.877237] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.877244] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.877506] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.877526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.877539] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.877550] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.877870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.877893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.877914] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=19][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.877922] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.878180] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.878199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.878211] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.878219] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.878449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.878470] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.878482] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.878490] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.878661] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.878679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.878691] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.878700] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.878918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.878932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.878941] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.878947] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.879019] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.879031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.879038] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.879049] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.879105] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.879120] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.879127] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.879132] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.879408] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.879424] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.879433] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.879439] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.879869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.879886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.879900] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=14][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.879906] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.880156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880174] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880182] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.880189] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.880395] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880418] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.880424] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.880531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880562] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=9][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.880568] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.880676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880689] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880697] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.880703] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.880820] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880833] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880841] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.880848] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.880964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880979] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.880987] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.880992] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.881111] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881123] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881129] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.881134] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.881239] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881257] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.881267] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.881374] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881394] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.881399] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.881504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881516] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.881521] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.881623] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881635] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.881641] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.881760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881771] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.881778] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.881913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.881926] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.881933] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.882100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.882108] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.882114] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.882257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.882276] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.882290] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.882385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.882398] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.882404] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.882514] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.882536] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=21][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.882851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.882864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.882872] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.882877] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.883112] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.883133] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.883140] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.883301] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.883308] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.883439] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.883451] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.883457] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.883528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.883536] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.883547] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.883624] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.883636] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.883756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.883770] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.883777] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.883900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.883907] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.883926] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=19][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.884088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.884100] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.884106] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.884249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.884255] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.884266] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.884314] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.884482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.884494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.884503] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.884508] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.884707] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.884724] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.884731] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.884983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.884996] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.885019] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=22][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.885033] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.885198] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.885212] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.885181] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.885364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.885380] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.885386] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.885493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.885507] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.885513] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.885613] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.885625] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.885878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.885894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.885903] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.885908] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.885978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.885993] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.886000] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.886093] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.886107] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.886183] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.886197] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.886204] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.886289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.886303] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.886309] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.886519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.886537] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.886543] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.886665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.886677] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.886691] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=14][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.886875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.886903] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.886910] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.887132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.887144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.887153] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=9][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.887158] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.887529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.887543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.887552] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.887565] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.887699] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.887720] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.887726] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.887864] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.887881] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.887888] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888000] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.888032] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888039] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888115] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.888134] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888141] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888211] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888222] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888337] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.888349] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888355] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888470] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.888484] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888491] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888561] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.888574] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888585] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888697] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888710] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.888786] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888790] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=3][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888863] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888871] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.888917] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.888925] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.888929] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.889023] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.889031] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.889112] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.889120] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.889124] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.889249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.889259] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.889263] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.889362] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.889370] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.889499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.889513] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.889517] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.889652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.889661] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.889666] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.889753] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.889761] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.889765] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.889896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.889906] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1001, wash_size=6291456) [2025-04-03 14:01:19.889911] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B60-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:19.890377] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.890405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.890524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.890853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.891496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.891672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.891827] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.892021] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.892339] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.892798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.893177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.893192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.893338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.893482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.893629] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.893797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.893961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.894125] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.894267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.894431] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.895077] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.895260] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.895896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.896052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.896219] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.896404] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D6F-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.896891] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.897101] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.897583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.897692] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.898046] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.898328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.898857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.899020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.899313] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.899820] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.900167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.900316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.900448] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B61-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.901948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.902400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.902920] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.903061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.903577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.903729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.903887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.904041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.904506] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.904903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.905025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.905185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.905585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.905725] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.905835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.906073] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.906167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.907257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.907413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.907728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.907941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.908162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.908281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.908425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.908824] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.909422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.909733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.909900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.910222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.910724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.911369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.911531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B62-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.913242] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.913270] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=24][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.913281] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.913305] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.920271] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D71-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.920329] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D71-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1492, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.922587] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:19.931185] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D72-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.931214] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D72-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=824, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.941920] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D73-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.941950] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D73-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=634, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.952444] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D74-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.952470] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D74-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=414, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.962540] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=43] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:19.962946] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D75-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.962977] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D75-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=373, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.965638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.965667] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.965866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.965885] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.966070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.966088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.966388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.966407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.966604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.966622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.966802] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.966846] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.967036] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.967054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.967208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.967225] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.967479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.967519] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.967939] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.967958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.968582] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=45] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660075959581, last_first_register_time_us=1743648246543257, register_time_us=1743660075959581, curr_time_us=1743660079968578) [2025-04-03 14:01:19.968799] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.968856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.969528] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=53][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:19.969550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.969568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.969569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=36] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.969590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.969835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.969861] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.969868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.969888] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.970136] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.970169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.970182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.970199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.970217] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=38] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}, {id:1002}]) [2025-04-03 14:01:19.970477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.970492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.970729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.970754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.971775] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.971929] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4814886476, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:19.971990] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=53] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:19.972233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.972252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.972679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.972709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.973296] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4323, tid:17484}]) [2025-04-03 14:01:19.973494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.973514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.974194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.974213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.974407] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.974439] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.974458] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660079972390], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.974494] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=34][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.4:2882", msg={tenant_id:1, srr:[mts=1743660079972390], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.974536] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660079972390]}) [2025-04-03 14:01:19.975080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.975109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.975405] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.975430] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=23][errcode=-4013] oops, alloc failed, tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=9223372036854775807, tenant_hold=8096894976, tenant_limit=9223372036854775807 [2025-04-03 14:01:19.975475] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=37][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.975491] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=15][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:19.975505] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=12][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:19.975519] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=12][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.975548] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=20][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:19.975603] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=54] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.975640] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=36][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.975683] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.975676] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=35] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2523, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.975703] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.975717] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660079974563], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.975736] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1001, srr:[mts=1743660079974563], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.975712] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D76-0-0] [lt=34][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.975760] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660079974563]}) [2025-04-03 14:01:19.976203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.976238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.976421] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.976441] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.976702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.976729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977048] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977072] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977676] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.977689] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977710] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.977725] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660079975781], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.977741] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1002, srr:[mts=1743660079975781], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.977802] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660079975781]}) [2025-04-03 14:01:19.977919] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.977989] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:19.978037] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=45][errcode=-4013] oops, alloc failed, tenant_id=1003, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=1073741800, tenant_hold=441917440, tenant_limit=1073741824 [2025-04-03 14:01:19.978057] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=13][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_LOG_REGISTER_PARENT_REQ, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.978069] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=12][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.978083] WDIAG [PALF] post_request_to_server_ (log_net_service.h:306) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=10][errcode=-4013] LogRpc post_request failed(ret=-4013, palf_id_=1, req={child:{server:"172.20.0.5:2882", region:"default_region", register_time_us:1743660079969364, keepalive_ts:-1}, is_to_leader:true}, server="172.20.0.3:2882") [2025-04-03 14:01:19.978116] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=31] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.20.0.5:2882") [2025-04-03 14:01:19.978142] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.978954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.978978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979502] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.979530] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.979544] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660079977852], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.979569] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1003, srr:[mts=1743660079977852], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.979590] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660079977852]}) [2025-04-03 14:01:19.979602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979687] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979815] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979826] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.979940] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980212] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980599] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980870] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980879] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.980993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981158] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_GET_GTS_REQUEST, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.981194] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:19.981206] WDIAG [STORAGE.TRANS] post (ob_gts_rpc.cpp:248) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", server="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660079980021], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.981228] WDIAG [STORAGE.TRANS] query_gts_ (ob_gts_source.cpp:562) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4013] post gts request failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", leader="172.20.0.3:2882", msg={tenant_id:1004, srr:[mts=1743660079980021], range_size:1, sender:"172.20.0.5:2882"}) [2025-04-03 14:01:19.981264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981251] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660079980021]}) [2025-04-03 14:01:19.981474] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981483] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981577] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981629] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981861] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.981981] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982082] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982265] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.982276] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=10][errcode=-4013] rpc memory pool alloc memory failed(sz=1340, alloc_sz=16256) [2025-04-03 14:01:19.982281] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=5][errcode=-4013] alloc buffer fail(payload_sz=1156) [2025-04-03 14:01:19.982290] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.982299] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=997699, timeout_timestamp=1743660080977864) [2025-04-03 14:01:19.982308] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:19.982313] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:19.982319] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:19.982324] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=3][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:19.982329] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=3][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:19.982334] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:19.982343] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:19.982350] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=6][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:19.982373] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=15][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:19.982385] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=12][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:19.982389] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:19.982493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982502] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982827] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982929] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.982941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983327] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983465] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983724] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983732] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983802] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983824] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983923] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.983992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984120] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984313] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984377] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=95] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984491] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:19.984619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984727] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984968] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.984976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985075] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985159] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985416] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985609] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985616] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985700] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.985985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.986134] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.986144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.986270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.986279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.986479] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.986786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.986925] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.987497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.987500] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D77-0-0] [lt=43][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.987538] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D77-0-0] [lt=37][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:19.987549] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D77-0-0] [lt=10][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:19.987560] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D77-0-0] [lt=9][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.987579] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.987575] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D77-0-0] [lt=9][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:19.987615] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D77-0-0] [lt=1][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.987664] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D77-0-0] [lt=1][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.987857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.987867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.988071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.988215] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.988461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.988683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.988852] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.989027] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.989354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.989680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.989855] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:19.989866] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B63-0-0] [lt=10][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:19.989884] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=15][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:19.989890] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=5][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:19.989897] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=6][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:19.989903] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=5][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:19.989909] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=5][errcode=-4013] query failed(ret=-4013, conn=0x2b739ba2a050, start=1743660079977870, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:19.989919] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:19.989973] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=11][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:19.989980] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=6][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1002, ls_ids=[{id:1}]) [2025-04-03 14:01:19.989989] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=8][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1002, ls_ids=[{id:1}]) [2025-04-03 14:01:19.989996] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=6][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1002, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660079977768}) [2025-04-03 14:01:19.990016] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=19][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:19.990021] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B63-0-0] [lt=4][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:19.990399] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4682404476, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:19.990427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:19.990893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.990907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.990955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.990963] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.991121] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.991471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.991730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.991836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.991956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.992604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.992786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.993029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.993112] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.993353] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.993364] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=10][errcode=-4013] rpc memory pool alloc memory failed(sz=1374, alloc_sz=16256) [2025-04-03 14:01:19.993369] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=5][errcode=-4013] alloc buffer fail(payload_sz=1190) [2025-04-03 14:01:19.993379] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=9][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.993389] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=3][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1003, svr="172.20.0.3:2882", timeout=998532, timeout_timestamp=1743660080990171) [2025-04-03 14:01:19.993397] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=7][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:19.993402] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=3][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:19.993406] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=3][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:19.993410] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=3][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:19.993414] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=3][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:19.993418] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=3][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:19.993424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=5][errcode=-4013] execute failed(ret=-4013, tenant_id=1003, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:19.993430] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=5][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:19.993447] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=12][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:19.993459] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=12][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:19.993464] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=5][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:19.993509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.993743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.994063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.994281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.994408] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.994551] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.994744] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.995086] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.995190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.995551] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.995719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.995852] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.996039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.996353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.996361] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.996725] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.996841] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.997099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.997235] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.997365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.997481] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.997659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.997800] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.998042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.998142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.998296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.998413] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D78-0-0] [lt=34][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:19.998436] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D78-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:19.998445] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D78-0-0] [lt=8][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:19.998453] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D78-0-0] [lt=7][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:19.998473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.998464] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D78-0-0] [lt=7][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:19.998506] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D78-0-0] [lt=1][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:19.998538] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D78-0-0] [lt=0][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:19.998661] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.998774] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.999130] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.999334] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:19.999394] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:19.999403] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B64-0-0] [lt=0][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:19.999416] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=12][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:19.999423] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=5][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:19.999428] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=4][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1003) [2025-04-03 14:01:19.999432] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=4][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1003) [2025-04-03 14:01:19.999436] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=3][errcode=-4013] query failed(ret=-4013, conn=0x2b7322070050, start=1743660079990180, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:19.999443] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=6][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:19.999480] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=5][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004) [2025-04-03 14:01:19.999489] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=8][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:19.999496] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=6][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:19.999501] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=4][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1004, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660079981241}) [2025-04-03 14:01:19.999515] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=13][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:19.999519] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B64-0-0] [lt=4][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:20.009171] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=23][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:20.009195] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=23][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:20.009203] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=8][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:20.009210] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=6][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:20.009219] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=5][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:20.009251] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=31] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.009271] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=19][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.009308] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=36] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=643, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.009327] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D79-0-0] [lt=18][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.010017] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [17844][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=38] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:20.013313] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.013339] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.013359] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.013372] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7] [PALF_DUMP](palf_id=1003, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1003, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:615432, max_lsn:74920351, max_scn:{val:1743655479584546174, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:74920351}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:74671177}}, last_submit_lsn:{lsn:74920229}, last_submit_end_lsn:{lsn:74920351}, last_submit_log_id:615432, last_submit_log_pid:14, max_flushed_lsn:{lsn:74920229}, max_flushed_end_lsn:{lsn:74920351}, max_flushed_log_pid:14, committed_end_lsn:{lsn:74920229}, last_slide_log_id:615431, last_slide_scn:{val:1743655479483236415, v:0}, last_slide_lsn:{lsn:74920107}, last_slide_end_lsn:{lsn:74920229}, last_slide_log_pid:14, last_slide_log_accum_checksum:2102179633, last_fetch_end_lsn:{lsn:116863269}, last_fetch_max_log_id:617479, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660078612742, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7372ff0530}, [StateMgr]={this:0x2b7372ff4a30, self:"172.20.0.5:2882", palf_id:1003, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:17}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:74671289}, scan_disk_log_finished:true, last_check_start_id:615432, is_changing_config_with_arb:false, reconfirm_start_time_us:1743648262361526, palf_role_change_cb:0x2b7372ffb2f0, allow_vote:true}, [ConfigMgr]={palf_id:1003, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:14, prev:{version:1, config:{config_version:{proposal_id:13, config_seq:32}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:25, lock_time:1743614587856765}}, curr:{version:1, config:{config_version:{proposal_id:14, config_seq:33}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:25, lock_time:1743614587856765}}, prev_log_proposal_id:13, prev_lsn:{lsn:74671055}, prev_mode_pid:14}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:12, prev_lsn:{lsn:67040656}, prev_end_lsn:{lsn:67040778}, prev_mode_pid:13}, reconfig_barrier:{prev_log_proposal_id:12, prev_lsn:{lsn:67040656}, prev_end_lsn:{lsn:67040778}, prev_mode_pid:13}, persistent_config_version:{proposal_id:14, config_seq:33}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660078912237, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660078912237, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b7372ff1c70}, [ModeMgr]={palf_id:1003, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:14, mode_version:0, access_mode:1, ref_scn:{val:1743590085102110281, v:0}}, accepted_mode_meta:{version:1, proposal_id:14, mode_version:0, access_mode:1, ref_scn:{val:1743590085102110281, v:0}}, last_submit_mode_meta:{version:1, proposal_id:14, mode_version:0, access_mode:1, ref_scn:{val:1743590085102110281, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1003, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:0}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:17}, log_config_meta:{version:3, proposal_id:14, prev:{version:1, config:{config_version:{proposal_id:13, config_seq:32}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:25, lock_time:1743614587856765}}, curr:{version:1, config:{config_version:{proposal_id:14, config_seq:33}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:25, lock_time:1743614587856765}}, prev_log_proposal_id:13, prev_lsn:{lsn:74671055}, prev_mode_pid:14}, log_snapshot_meta:{version:1, base_lsn:{lsn:0}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:0, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:14, mode_version:0, access_mode:1, ref_scn:{val:1743590085102110281, v:0}}}, log_meta_storage:{log_tail:{lsn:49152}, readable_log_tail:{lsn:49152}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:134209536}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1003}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1003/meta", dir_fd:125, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67055616, block_header_serialize_buf_:0x2b7372ff6bb8, flashback_version:0}, log_storage:{log_tail:{lsn:74920351}, readable_log_tail:{lsn:74920351}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:67104768}, min_scn:{val:1743648327251630861, v:0}, curr_block_id:1, palf_id:1003}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1004/1003/log", dir_fd:126, min_block_id:0, max_block_id:2, curr_writable_block_id:1}, logical_block_size_:67104768, curr_block_writable_size_:59289185, block_header_serialize_buf_:0x2b7372ff8838, flashback_version:0}, palf_epoch:8, last_purge_throttling_ts:1743648262361055, this:0x2b7372ff5170}, [Reconfirm]={palf_id:1003, self:"172.20.0.5:2882", state:"FINISHED", new_proposal_id:13, prepare_log_ack_list:2"172.20.0.4:2882";"172.20.0.3:2882";, curr_paxos_follower_list:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, majority_cnt:2, majority_max_log_server:"172.20.0.5:2882", majority_max_accept_pid:12, majority_max_lsn:{lsn:67040778}, saved_end_lsn:{lsn:67040778}, last_submit_prepare_req_time_us:1743648262362091, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b7372ff4bb0}) [2025-04-03 14:01:20.013617] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=237] [PALF_DUMP](palf_id=1003, self="172.20.0.5:2882", ack_info_list=[{member:{server:"172.20.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671055}}, {member:{server:"172.20.0.4:2882", timestamp:1, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671055}}, {member:{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671289}}]) [2025-04-03 14:01:20.013637] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4308) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17] current log_task status(palf_id=1003, self="172.20.0.5:2882", log_id=615432, log_task={header:{begin_lsn:{lsn:74920229}, end_lsn:{lsn:74920351}, log_id:615432, min_scn:{val:1743655479584546174, v:0}, max_scn:{val:1743655479584546174, v:0}, data_len:66, proposal_id:14, prev_lsn:{lsn:74920107}, prev_proposal_id:14, committed_end_lsn:{lsn:74920229}, data_checksum:940009244, accum_checksum:2166578877, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1743659145927011, freeze_ts:1743659145927011, submit_ts:1743659145927013, flushed_ts:1743659145932829, gen_to_freeze cost time:0, gen_to_submit cost time:2, submit_to_flush cost time:5816}, ack_info_list=[{member:{server:"172.20.0.3:2882", timestamp:1, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671055}}, {member:{server:"172.20.0.4:2882", timestamp:1, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671055}}, {member:{server:"172.20.0.5:2882", timestamp:1743648246150910, flag:0}, last_ack_time_us:1743655269116743, last_flushed_end_lsn:{lsn:74671289}}]) [2025-04-03 14:01:20.013671] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.019800] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7A-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.019855] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7A-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=362, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.022662] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.030688] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.030762] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7B-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=797, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.041620] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7C-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.041706] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7C-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=760, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.045546] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=54] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:20.046056] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=36] Wash time detail, (compute_wash_size_time=183, refresh_score_time=93, wash_time=15) [2025-04-03 14:01:20.051350] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5276, clean_start_pos=1069538, clean_num=62914) [2025-04-03 14:01:20.052410] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7D-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.052502] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7D-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=590, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.053885] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.053902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.053920] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054093] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054180] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054348] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054429] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054752] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.054854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055032] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055238] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055466] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055602] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055663] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055772] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.055994] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056158] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056228] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056281] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056584] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056656] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056663] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056837] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056881] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056922] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056972] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.056977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057048] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057149] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057193] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057197] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057241] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057246] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057353] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057358] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057466] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057568] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057616] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057664] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057722] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057730] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057890] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057939] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.057987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058042] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058155] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058204] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058212] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058257] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058268] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058317] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058470] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058522] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058572] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058817] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.058992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.059295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.059622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.059847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.059961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.060110] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B65-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.063524] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7E-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.063600] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7E-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=913, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.065605] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48] get wrs ts(ls_id={id:1}, delta=4814979731, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.065636] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.068667] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.073506] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:8262, tid:17514}]) [2025-04-03 14:01:20.075540] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7F-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.075610] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D7F-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1815, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.082387] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660080081828]}) [2025-04-03 14:01:20.082990] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.084299] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660080082437]}) [2025-04-03 14:01:20.084564] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.085612] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660080084338]}) [2025-04-03 14:01:20.086568] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D80-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.086633] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D80-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=777, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.086884] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660080085640]}) [2025-04-03 14:01:20.087286] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.087318] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.087328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.087632] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.087657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.087671] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.087829] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.087842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.087848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.087959] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.087985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.087991] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088172] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.088190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088259] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.088272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088609] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.088626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088632] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088759] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.088776] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088782] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.088984] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.088998] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089160] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.089175] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089181] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089273] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.089286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089291] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089380] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.089388] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=7] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089400] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089656] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.089681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089936] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.089950] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.089961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090064] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090076] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090290] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660080087275]}) [2025-04-03 14:01:20.090465] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090518] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090525] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090553] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090567] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090587] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090620] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090631] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090659] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090662] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090689] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090735] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.090747] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090802] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090859] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090889] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090895] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090916] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090962] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090963] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 3 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.090981] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091062] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.091114] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091144] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091236] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091309] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.091415] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091493] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091570] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091638] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091662] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=23] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.091825] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091902] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.091916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.091961] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092013] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092090] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092138] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092238] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.092296] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092389] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092496] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.092567] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092654] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092708] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092756] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.092859] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092912] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.092987] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093050] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.093127] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093209] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093285] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093349] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093360] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.093402] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093447] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093489] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093586] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093641] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=6] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.093692] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093746] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093802] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093858] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.093909] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094095] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.094159] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094203] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094309] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.094372] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094422] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094458] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094503] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094591] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.094650] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094697] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094749] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094822] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.094916] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.094974] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095034] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095082] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095134] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.095186] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095294] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095354] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095438] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.095493] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095544] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=1][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095588] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095643] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095691] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.095761] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095844] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095976] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.095987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.096110] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.096203] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.096249] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.096301] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.096311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.096353] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.096405] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.096453] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.096504] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B66-0-0] [lt=0][errcode=0] [CACHE] len: 1600 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 101 | get_cnt= 338 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 47 | get_cnt= 54 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 23 | get_cnt= 150 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 28 | get_cnt= 17771 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 46 | get_cnt= 17763 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 4 | get_cnt= 6 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 148 | get_cnt= 185 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 331 | get_cnt= 1623095 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 55 | get_cnt= 22488 | score=0.000000 | [CACHE] tenant_id= 1001 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 1 | get_cnt= 1 | score=0.000000 | [2025-04-03 14:01:20.097388] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D81-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.097469] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:20.097487] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.097494] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.097463] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D81-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=616, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.097563] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:20.097578] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.097630] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:20.097744] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:20.097758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.097863] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:20.097918] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:20.097969] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:20.098033] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:20.098043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.098336] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.098689] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.098845] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.099190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.099326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.099556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.099856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.100021] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.100137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.100654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.100843] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.100957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.101202] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.101620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.101803] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.102012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.102414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.102733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.102971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.103714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.103902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.104033] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.104227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.104385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.104530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B67-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.108915] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D82-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.109037] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D82-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1307, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.113333] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.113358] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.113369] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.113378] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.120163] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D83-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.120206] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D83-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=519, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.123857] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.128910] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.128936] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.128945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129078] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129284] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129303] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129490] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129766] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.129900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130068] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130192] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130285] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130420] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130569] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130674] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.130893] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.131224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.131251] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.131436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.131457] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.131626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.131648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.131831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.131851] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.132418] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D84-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.132512] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D84-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1946, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.134464] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.134493] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.134635] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.134647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.134920] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.134934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.135108] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.135119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.135382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.135417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.135550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.135560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136038] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136051] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136302] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136315] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136783] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.136958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.137514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.137529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.137830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.137857] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.138338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.138362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.138510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.138520] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.138640] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.138660] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.138797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.138806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.139265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.139279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.139333] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.139491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.139517] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.139773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.139784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.139915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.139932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140285] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140440] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140632] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.140798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141154] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141337] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141535] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141542] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141794] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141802] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.141934] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142719] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.142979] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.143394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.143476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.143636] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.143806] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.143978] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.144122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.144278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.144592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.144767] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.144921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.145070] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.145425] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.145884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.145899] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.146088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.146261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.146409] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.146537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.146698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.147088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.147352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DB-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.151714] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-04-03 14:01:20.151783] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=49] get_disk_usage(ret=0, capacity(MB):=9216, used(MB):=563) [2025-04-03 14:01:20.151884] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18622][T1004_CKClogDis][T1004][Y0-0000000000000000-0-0] [lt=58] cannot_recycle_log_size statistics(cannot_recycle_log_size=322182499, threshold=2899102924, need_update_checkpoint_scn=false) [2025-04-03 14:01:20.155025] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D86-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.155100] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D86-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1257, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.165987] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.166047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.166405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.166428] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.166892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.166914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.167188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.167562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.167701] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D87-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.167763] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D87-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2478, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.168032] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.168252] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.168434] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.168751] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=43][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.175434] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18325][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=33] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.176906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B69-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.176983] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B69-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.184638] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.189979] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:20.189997] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28738 [2025-04-03 14:01:20.190061] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=18] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28738 [2025-04-03 14:01:20.190068] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.190484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B6B-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.190532] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4682604604, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:20.191188] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D89-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.193328] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154E-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.194247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154E-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.194351] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [17821][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=51] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.195458] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.202603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B6D-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.203757] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154E-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.204017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154E-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.206985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154E-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.207461] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154E-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.210530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B6E-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.212901] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8B-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.212986] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8B-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=853, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.213437] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.213492] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.213512] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.213534] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.220023] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.220041] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28744 [2025-04-03 14:01:20.220081] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28744 [2025-04-03 14:01:20.220087] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.220214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B70-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.223956] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.224107] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8C-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.226723] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.226771] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28746 [2025-04-03 14:01:20.226871] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28746 [2025-04-03 14:01:20.226900] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.232364] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.232406] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28748 [2025-04-03 14:01:20.232488] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28748 [2025-04-03 14:01:20.232512] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.235404] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8D-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.235505] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8D-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1124, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.235728] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=1] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 438.00; ) [2025-04-03 14:01:20.235823] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS CONFIG](tenant_id=500, ref_cnt=2, io_config={group_num_:0, memory_limit_:536870912, callback_thread_count_:8, unit_config_:{min_iops:10000, max_iops:50000, weight:10000}, enable_io_tracer:false, group_configs:[other_groups:{deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, allocated_memory=10735648, pre_allocated_count=2, callback_queues=[0, 0, 0, 0, 0, 0, 0, 0]) [2025-04-03 14:01:20.236026] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=69] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:20.241357] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.241431] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=65] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28750 [2025-04-03 14:01:20.241511] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28750 [2025-04-03 14:01:20.241535] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.246791] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.246890] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8E-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1053, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.251878] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:20.252340] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash time detail, (compute_wash_size_time=203, refresh_score_time=111, wash_time=20) [2025-04-03 14:01:20.257424] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5055, clean_start_pos=1132452, clean_num=62914) [2025-04-03 14:01:20.257870] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8F-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.257962] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D8F-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=866, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.260460] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.260511] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.260785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.260932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.261077] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.261209] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.261411] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.261659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.261992] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.262305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.262459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.262658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.262914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.263077] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.263379] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.263545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.263654] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.263677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.263781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.263793] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.263993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.264040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.264217] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.264272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.264378] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.264419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.264582] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.264637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.264751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.265055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.265250] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.265416] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.265850] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4815180201, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.265917] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.266272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.266439] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.266665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.266844] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.267035] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.267267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.267500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.267658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.267828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.267973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.268216] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.268485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.268702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.268837] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.268933] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D90-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.268998] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D90-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=835, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.269170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.269407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.269683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.270037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.270386] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.270539] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE2FF-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.274905] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.274942] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28752 [2025-04-03 14:01:20.275041] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28752 [2025-04-03 14:01:20.275063] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.280060] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D91-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.280131] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D91-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=916, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.284714] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.290603] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.290626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290708] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290743] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290777] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290818] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290887] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290951] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.290986] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.291023] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.291065] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.291103] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.291138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.291169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.291200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.291245] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4682704771, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:20.291266] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.293372] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D92-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.293450] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D92-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2913, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.298671] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.298690] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28754 [2025-04-03 14:01:20.298720] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28754 [2025-04-03 14:01:20.298726] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.301131] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660080300018]}) [2025-04-03 14:01:20.302098] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660080301169]}) [2025-04-03 14:01:20.302561] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=6] ====== check clog disk timer task ====== [2025-04-03 14:01:20.302584] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=18] get_disk_usage(ret=0, capacity(MB):=1024, used(MB):=255) [2025-04-03 14:01:20.302608] INFO [STORAGE] cannot_recycle_log_over_threshold_ (ob_checkpoint_service.cpp:260) [18418][T1003_CKClogDis][T1003][Y0-0000000000000000-0-0] [lt=9] cannot_recycle_log_size statistics(cannot_recycle_log_size=200526433, threshold=322122547, need_update_checkpoint_scn=false) [2025-04-03 14:01:20.303124] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=3] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=7) [2025-04-03 14:01:20.303689] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660080302129]}) [2025-04-03 14:01:20.304546] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660080303708]}) [2025-04-03 14:01:20.304675] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D93-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.304734] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D93-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1082, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.305990] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660080304828]}) [2025-04-03 14:01:20.306437] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.306623] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.306760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.306883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.307035] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.307203] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.307601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.307913] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.308260] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.308280] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28756 [2025-04-03 14:01:20.308312] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28756 [2025-04-03 14:01:20.308319] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.309321] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.309539] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.309749] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.310026] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.310513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.310787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.311234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.311758] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.312028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B71-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.313517] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.313545] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=23][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.313558] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.313571] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.318303] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE37A-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.318448] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E5-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.324088] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.326802] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28758 [2025-04-03 14:01:20.328914] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [18497][T1004_PalfGC][T1004][Y0-0000000000000000-0-0] [lt=21] BlockGCTimerTask success(ret=0, cost_time_us=16, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.20.0.5:2882", log_dir:"/data/oceanbase_data/clog/tenant_1004", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):9216, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):9216, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):563, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-04-03 14:01:20.329895] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.332169] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E5-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.347964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B72-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.351917] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:20.358674] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D98-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.358750] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D98-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=621, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.363589] INFO [SHARE] refresh_primary_cluster_id (ob_alive_server_tracer.cpp:397) [17417][ServerTracerTim][T0][YB42AC140005-000631D6BBECBE96-0-0] [lt=21] refresh primary cluster finish(ret=0, primary_cluster_id_=-1) [2025-04-03 14:01:20.365977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.366279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.366311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.366482] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [18385][T1003_TenantMet][T1003][Y0-0000000000000000-0-0] [lt=36] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:754, free_obj_hold_cnt:90, allocator used:3355744, allocator total:3459840}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:77, free_obj_hold_cnt:6, allocator used:5444053, allocator total:5444053}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=8903893, tablets_mem_limit=214748360, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:16, free_obj_hold_cnt:46, allocator used:123008, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=831) [2025-04-03 14:01:20.366601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.366891] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.367167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.367412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.367605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.367754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.368059] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.368316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.368657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.368858] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.368873] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=33][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.369459] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.369795] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.370186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.370527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.370844] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D99-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.370925] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D99-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1972, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.371106] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.371622] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4815286614, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.371666] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.380636] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.380654] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28766 [2025-04-03 14:01:20.380693] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28766 [2025-04-03 14:01:20.380699] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.381735] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9A-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.381803] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9A-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=661, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.383710] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:20.383728] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=15] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28768 [2025-04-03 14:01:20.383758] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28768 [2025-04-03 14:01:20.383763] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.384799] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=28] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660076280985, last_first_register_time_us=1743648235473854, register_time_us=1743660076280985, curr_time_us=1743660080384795) [2025-04-03 14:01:20.385177] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=76][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.385223] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.385438] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.385585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.385738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.385910] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.386187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.386322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.386481] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.386711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.386975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.387161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.387389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.387583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.387896] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.388226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.388687] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.388734] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.20.0.5:2882") [2025-04-03 14:01:20.388757] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.390713] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=4682804630, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:20.390778] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=58] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.393033] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9B-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.393133] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9B-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1071, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.395611] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=2] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.395635] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=20] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28770 [2025-04-03 14:01:20.395671] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28770 [2025-04-03 14:01:20.395681] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.404317] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9C-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.404416] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9C-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=995, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.407425] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660080406535]}) [2025-04-03 14:01:20.408606] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660080407460]}) [2025-04-03 14:01:20.409486] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660080408630]}) [2025-04-03 14:01:20.410298] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660080409509]}) [2025-04-03 14:01:20.411683] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.411723] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=32] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28772 [2025-04-03 14:01:20.411800] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28772 [2025-04-03 14:01:20.411837] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660080410584]}) [2025-04-03 14:01:20.411849] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=44] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.413751] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=2][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.413803] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.413852] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=41][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.413872] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.424174] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.427805] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9E-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.431890] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28774 [2025-04-03 14:01:20.437858] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.437904] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28776 [2025-04-03 14:01:20.437975] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28776 [2025-04-03 14:01:20.437997] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.438453] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9F-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.438490] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3D9F-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=513, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.449015] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA0-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.449052] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA0-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=495, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.458114] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:20.458724] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=73] Wash time detail, (compute_wash_size_time=272, refresh_score_time=111, wash_time=19) [2025-04-03 14:01:20.461390] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=30] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.461419] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA1-0-0] [lt=0][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.465541] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.466187] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36] get wrs ts(ls_id={id:1}, delta=4815380450, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.468995] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=140 [2025-04-03 14:01:20.469477] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2699) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=46] [PALF_DUMP](palf_id=1, self="172.20.0.5:2882", [SlidingWindow]={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:40888393, max_lsn:23016225752, max_scn:{val:1743655278027912055, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:23016148094}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:22974282712}}, last_submit_lsn:{lsn:23016147553}, last_submit_end_lsn:{lsn:23016148094}, last_submit_log_id:40888245, last_submit_log_pid:620, max_flushed_lsn:{lsn:23016147553}, max_flushed_end_lsn:{lsn:23016148094}, max_flushed_log_pid:620, committed_end_lsn:{lsn:23016147553}, last_slide_log_id:40888244, last_slide_scn:{val:1743655265673135133, v:0}, last_slide_lsn:{lsn:23016147012}, last_slide_end_lsn:{lsn:23016147553}, last_slide_log_pid:620, last_slide_log_accum_checksum:2388580578, last_fetch_end_lsn:{lsn:23058090593}, last_fetch_max_log_id:40890292, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660078675180, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b734c5f2530}, [StateMgr]={this:0x2b734c5f6a30, self:"172.20.0.5:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:620}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:40888245, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x2b734c5fd2f0, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"172.20.0.5:2882", alive_paxos_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246040918, flag:0}, alive_paxos_replica_num:3, log_ms_meta:{version:3, proposal_id:620, prev:{version:1, config:{config_version:{proposal_id:620, config_seq:404}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246040918, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:620, config_seq:405}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246040918, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:620, prev_lsn:{lsn:22954617481}, prev_mode_pid:620}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:620, config_seq:405}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", paxos_member_region_map:{}, register_time_us:1743660079969364, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:1743660079969364, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x2b734c5f3c70}, [ModeMgr]={palf_id:1, self:"172.20.0.5:2882", applied_mode_meta:{version:1, proposal_id:620, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:620, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:620, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, base_lsn_for_block_gc:{lsn:22815621120}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:620}, log_config_meta:{version:3, proposal_id:620, prev:{version:1, config:{config_version:{proposal_id:620, config_seq:404}, log_sync_memberlist:2{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:1, learner_array:[{server:"172.20.0.5:2882", timestamp:1743648246040918, flag:2}]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:620, config_seq:405}, log_sync_memberlist:3{server:"172.20.0.3:2882", timestamp:1, flag:0}{server:"172.20.0.4:2882", timestamp:1, flag:0}{server:"172.20.0.5:2882", timestamp:1743648246040918, flag:0}, log_sync_replica_num:3, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:620, prev_lsn:{lsn:22954617481}, prev_mode_pid:620}, log_snapshot_meta:{version:1, base_lsn:{lsn:22815621120}, prev_log_info:{log_id:40553300, lsn:{lsn:22815616575}, scn:{val:1743633390740619491, v:0}, log_proposal_id:620, accum_checksum:327345567}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:620, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:24576}, readable_log_tail:{lsn:24576}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:23016935424}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1003/1/meta", dir_fd:111, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67080192, block_header_serialize_buf_:0x2b734c5f8bb8, flashback_version:0}, log_storage:{log_tail:{lsn:23016148094}, readable_log_tail:{lsn:23016148094}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:22949830656}, min_scn:{val:1743647962249965295, v:0}, curr_block_id:342, palf_id:1}, block_mgr:{log_dir:"/data/oceanbase_data/clog/tenant_1003/1/log", dir_fd:112, min_block_id:340, max_block_id:343, curr_writable_block_id:342}, logical_block_size_:67104768, curr_block_writable_size_:787330, block_header_serialize_buf_:0x2b734c5fa838, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:1743648274772172, this:0x2b734c5f7170}, [Reconfirm]={palf_id:1, self:"172.20.0.5:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x2b734c5f6bb0}) [2025-04-03 14:01:20.469805] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=319][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:20.469866] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2703) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=59] [PALF_DUMP](palf_id=1, self="172.20.0.5:2882", ack_info_list=[]) [2025-04-03 14:01:20.469880] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2759) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="172.20.0.5:2882", tmp_server={server:"172.20.0.3:2882", timestamp:1, flag:0}) [2025-04-03 14:01:20.469889] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4308) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=8] current log_task status(palf_id=1, self="172.20.0.5:2882", log_id=40888245, log_task={header:{begin_lsn:{lsn:23016147553}, end_lsn:{lsn:23016148094}, log_id:40888245, min_scn:{val:1743655265844129995, v:0}, max_scn:{val:1743655265844129995, v:0}, data_len:485, proposal_id:620, prev_lsn:{lsn:23016147012}, prev_proposal_id:620, committed_end_lsn:{lsn:23016147553}, data_checksum:2093579608, accum_checksum:2415177702, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1743655265844942, freeze_ts:1743655265844942, submit_ts:1743655265844953, flushed_ts:1743655265846967, gen_to_freeze cost time:0, gen_to_submit cost time:11, submit_to_flush cost time:2014}, ack_info_list=[]) [2025-04-03 14:01:20.469919] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.476411] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B75-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.483741] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.483777] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28784 [2025-04-03 14:01:20.483871] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28784 [2025-04-03 14:01:20.483890] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.484428] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA3-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.484488] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA3-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=590, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.484866] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.487335] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.487365] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=25] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28786 [2025-04-03 14:01:20.487420] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28786 [2025-04-03 14:01:20.487436] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.490168] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE7F-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.491421] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18186][T1002_FreInfoRe][T1001][YB42AC140005-000631D6C3FCEE7F-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.496228] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA4-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.496285] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA4-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=674, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.496356] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:20.496479] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28788 [2025-04-03 14:01:20.500691] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18186][T1002_FreInfoRe][T1002][YB42AC140005-000631D6C3FCEE80-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.503998] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=33] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=6) [2025-04-03 14:01:20.504120] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:20.504167] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28792 [2025-04-03 14:01:20.504254] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28792 [2025-04-03 14:01:20.504281] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.504894] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [18600][T1004_LockWaitM][T1004][Y0-0000000000000000-0-0] [lt=36] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-04-03 14:01:20.506933] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA5-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.507058] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA5-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=633, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.513159] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.513255] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660080512452]}) [2025-04-03 14:01:20.513858] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.513981] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=48][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.514564] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660080513293]}) [2025-04-03 14:01:20.514979] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.515032] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] do_fetch_log_ finished(ret=0, palf_id=1002, self="172.20.0.5:2882", dest="172.20.0.4:2882", fetch_log_count=2048, fetch_start_lsn={lsn:74011080}, prev_lsn={lsn:74010958}, fetch_start_log_id=607966, last_slide_log_id=607965, fetch_log_size=41943040, accepted_mode_pid=1, last_fetch_req_time=1743660080515030, last_fetch_end_lsn={lsn:115954120}, last_fetch_max_log_id=610013, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1002, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:608085, max_lsn:74025692, max_scn:{val:1743655277950177453, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:74011080}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:32082652}}, last_submit_lsn:{lsn:74010958}, last_submit_end_lsn:{lsn:74011080}, last_submit_log_id:607965, last_submit_log_pid:1, max_flushed_lsn:{lsn:74010958}, max_flushed_end_lsn:{lsn:74011080}, max_flushed_log_pid:1, committed_end_lsn:{lsn:74011080}, last_slide_log_id:607965, last_slide_scn:{val:1743655265068231480, v:0}, last_slide_lsn:{lsn:74010958}, last_slide_end_lsn:{lsn:74011080}, last_slide_log_pid:1, last_slide_log_accum_checksum:1277512058, last_fetch_end_lsn:{lsn:115954120}, last_fetch_max_log_id:610013, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660080515030, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b73ab3c0530}) [2025-04-03 14:01:20.515136] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=98][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.515158] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.515174] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.515738] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660080514604]}) [2025-04-03 14:01:20.516948] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.517028] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=71] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28794 [2025-04-03 14:01:20.517103] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28794 [2025-04-03 14:01:20.517139] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=31] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.517361] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660080515770]}) [2025-04-03 14:01:20.517728] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA6-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.517786] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA6-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=523, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.519115] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18292][T1003_Occam][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.519234] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660080517746]}) [2025-04-03 14:01:20.524252] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=41][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.528987] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA7-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.530757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B76-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.535199] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18292][T1003_Occam][T1][YB42AC140005-000631D6BFBCDA1E-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.540190] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA8-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.540294] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DA8-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=976, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.548382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B77-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.561635] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:20.561661] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28796 [2025-04-03 14:01:20.561714] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28796 [2025-04-03 14:01:20.561728] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.564684] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAA-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.564771] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAA-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2247, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.569626] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:732) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-04-03 14:01:20.569695] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1036) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=58] get_min_unreplayed_log_info(lsn={lsn:13483027323}, scn={val:1743655265084940963, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1743655265088115, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:12746101, ret_code_:0}}, next_to_submit_lsn_:{lsn:13483027323}, next_to_submit_scn_:{val:1743655265084940963, v:0}, base_lsn_:{lsn:13286744064}, base_scn_:{val:1743616801173178850, v:0}, iterator_:{iterator_impl:{buf_:0x2b738c205000, next_round_pread_size:2121728, curr_read_pos:537, curr_read_buf_start_pos:0, curr_read_buf_end_pos:537, log_storage_:{IteratorStorage:{start_lsn:{lsn:13483026786}, end_lsn:{lsn:13483027323}, read_buf:{buf_len_:2125824, buf_:0x2b738c205000}, block_size:67104768, log_storage_:0x2b7398e6fbf0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1743655265084940962, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:449, scn_:{val:1743655265084940962, v:0}, data_checksum:2328545553, flag:1}}, init_mode_version:0, accumulate_checksum:1426087181, curr_entry_is_padding:0, padding_entry_size:481, padding_entry_scn:{val:1743655265084940962, v:0}}}}}) [2025-04-03 14:01:20.569830] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=129] get wrs ts(ls_id={id:1}, delta=4815483944, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.569843] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.569857] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.574298] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:103) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=0] tx gc loop thread is running(MTL_ID()=1) [2025-04-03 14:01:20.574364] INFO [STORAGE] update_calc_upper_info_ (ob_tx_data_table.cpp:1084) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=14] update calc upper info once(calc_upper_info_={min_start_scn_in_ctx_:{val:1743655261104510952, v:0}, keep_alive_scn_:{val:1743655261104510952, v:0}, update_ts_:1743660080574362}) [2025-04-03 14:01:20.575339] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.575357] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28798 [2025-04-03 14:01:20.575396] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28798 [2025-04-03 14:01:20.575406] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.575921] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:238) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616801173178850, v:0}, clog_base_lsn:{lsn:13286744064}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1743618797082332550, v:0}, all_id_meta:{id_meta:[{limited_id:1743655279117594735, latest_log_ts:{val:1743655262469466896, v:0}}, {limited_id:49000001, latest_log_ts:{val:1743605935907428857, v:0}}, {limited_id:360000001, latest_log_ts:{val:1743640357777174079, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:159, palf_env_:0x2b730a3fc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:7936, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}) [2025-04-03 14:01:20.577140] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAB-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.577227] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAB-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1040, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.581157] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.581190] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=28] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28800 [2025-04-03 14:01:20.581234] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28800 [2025-04-03 14:01:20.581246] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.584945] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.588978] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAC-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.589098] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAC-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1692, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.590845] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4683004515, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:20.590868] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.594380] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.594400] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28802 [2025-04-03 14:01:20.594435] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28802 [2025-04-03 14:01:20.594445] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.595410] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=34][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:20.595442] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=30][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660105653097, now=[mts=1743660079370681], now0=[mts=1743660079370681], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:20.595492] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=46][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b73a431d2a0, tx_id:{txid:76175015}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660079363941, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:20.595568] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=69][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b739ba5c1a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a431d2a0}) [2025-04-03 14:01:20.595604] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=34][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b73a431d2a0, tx_id:{txid:76175015}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660079363941, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:20.595661] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=54] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f8054008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660079370358, tx_desc={this:0x2b73a431d2a0, tx_id:{txid:76175015}, state:1, addr:"172.20.0.5:2882", tenant_id:1003, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660079363941, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999984, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:true, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660075723113, use_das=false, nested_level=0, session={this:0x2b739ba5c1a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1003, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b73a431d2a0}, plan=0x2b7338b48050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:20.595763] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=100][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737616891682840, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:20.595788] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=22][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:20.595801] WDIAG [SQL] open (ob_result_set.cpp:158) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=11][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:20.595832] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=28][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:20.595851] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=16][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:20.595870] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=16][errcode=-4283] execute failed(ret=-4283, tenant_id=1003, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=3, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:20.595892] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=18] will sleep(sleep_us=3000, remain_us=25127206, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=3, v.err_=-4283, timeout_timestamp=1743660105723097) [2025-04-03 14:01:20.599123] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=39][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:3, local_retry_times:3, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:20.599190] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=43][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:20.599203] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=12][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:20.599213] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=9][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:20.599308] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=20][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.600295] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAD-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.600352] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAD-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1067, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.601214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.602503] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.602522] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28804 [2025-04-03 14:01:20.602530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.602555] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28804 [2025-04-03 14:01:20.602561] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.602725] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.602765] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1004, 1003, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', 0, 0, 'NORMAL', 0, 100, 1006, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648246150910', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 1006, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:20.604310] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17486][LSUserTblUp0][T1003][YB42AC140005-000631D6BD2CC34E-0-0] [lt=0][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.3:2882") [2025-04-03 14:01:20.605379] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:66) [18220][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=1731] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-04-03 14:01:20.610755] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=3] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.610789] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=29] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28806 [2025-04-03 14:01:20.610841] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28806 [2025-04-03 14:01:20.610848] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.611462] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAE-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.611542] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAE-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=819, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.614135] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.614160] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=21][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.614178] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.614533] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1] do_fetch_log_ finished(ret=0, palf_id=1003, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:74920229}, prev_lsn={lsn:74920107}, fetch_start_log_id=615432, last_slide_log_id=615431, fetch_log_size=41943040, accepted_mode_pid=14, last_fetch_req_time=1743660080614532, last_fetch_end_lsn={lsn:116863269}, last_fetch_max_log_id=617479, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1003, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:615432, max_lsn:74920351, max_scn:{val:1743655479584546174, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:74920351}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:74671177}}, last_submit_lsn:{lsn:74920229}, last_submit_end_lsn:{lsn:74920351}, last_submit_log_id:615432, last_submit_log_pid:14, max_flushed_lsn:{lsn:74920229}, max_flushed_end_lsn:{lsn:74920351}, max_flushed_log_pid:14, committed_end_lsn:{lsn:74920229}, last_slide_log_id:615431, last_slide_scn:{val:1743655479483236415, v:0}, last_slide_lsn:{lsn:74920107}, last_slide_end_lsn:{lsn:74920229}, last_slide_log_pid:14, last_slide_log_accum_checksum:2102179633, last_fetch_end_lsn:{lsn:116863269}, last_fetch_max_log_id:617479, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660080614532, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b7372ff0530}) [2025-04-03 14:01:20.614601] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=65][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.620873] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.620944] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660080619987]}) [2025-04-03 14:01:20.620988] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:171) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", gts_tenant_info={v:1}) [2025-04-03 14:01:20.622398] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAF-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.622467] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DAF-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=721, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.623221] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660080621037]}) [2025-04-03 14:01:20.624314] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=49][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.625117] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660080623275]}) [2025-04-03 14:01:20.627088] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B78-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.627433] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660080625147]}) [2025-04-03 14:01:20.635242] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.643958] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B79-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.650497] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.655648] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB2-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.655745] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB2-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=892, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.666099] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:20.666397] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=4815580262, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.666412] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=34] Wash time detail, (compute_wash_size_time=203, refresh_score_time=74, wash_time=11) [2025-04-03 14:01:20.666427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.667135] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB3-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.667224] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB3-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1286, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.669904] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=3480, clean_start_pos=1258280, clean_num=62914) [2025-04-03 14:01:20.674393] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:110) [17845][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=106] try gc retain ctx [2025-04-03 14:01:20.675063] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.675086] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28810 [2025-04-03 14:01:20.675177] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28810 [2025-04-03 14:01:20.675192] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.675312] INFO [PALF] do_fetch_log_ (log_sliding_window.cpp:2125) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=0] do_fetch_log_ finished(ret=0, palf_id=1, self="172.20.0.5:2882", dest="172.20.0.3:2882", fetch_log_count=2048, fetch_start_lsn={lsn:23016147553}, prev_lsn={lsn:23016147012}, fetch_start_log_id=40888245, last_slide_log_id=40888244, fetch_log_size=41943040, accepted_mode_pid=620, last_fetch_req_time=1743660080675311, last_fetch_end_lsn={lsn:23058090593}, last_fetch_max_log_id=40890292, last_fetch_committed_end_lsn={lsn:18446744073709551615}, trigger_type="LOG_LOOP_TH", this={palf_id:1, self:"172.20.0.5:2882", lsn_allocator:{max_log_id:40888393, max_lsn:23016225752, max_scn:{val:1743655278027912055, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:0}, reuse_lsn:{lsn:23016148094}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:22974282712}}, last_submit_lsn:{lsn:23016147553}, last_submit_end_lsn:{lsn:23016148094}, last_submit_log_id:40888245, last_submit_log_pid:620, max_flushed_lsn:{lsn:23016147553}, max_flushed_end_lsn:{lsn:23016148094}, max_flushed_log_pid:620, committed_end_lsn:{lsn:23016147553}, last_slide_log_id:40888244, last_slide_scn:{val:1743655265673135133, v:0}, last_slide_lsn:{lsn:23016147012}, last_slide_end_lsn:{lsn:23016147553}, last_slide_log_pid:620, last_slide_log_accum_checksum:2388580578, last_fetch_end_lsn:{lsn:23058090593}, last_fetch_max_log_id:40890292, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:1743660080675311, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x2b734c5f2530}) [2025-04-03 14:01:20.675412] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=92][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.678620] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB4-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.678909] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB4-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1324, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.679424] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [17830][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=0] ====== traversal_flush timer task ====== [2025-04-03 14:01:20.679806] INFO [MDS] flush (mds_table_mgr.cpp:172) [17830][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=40] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b7398e04510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616801173178850, v:0}, clog_base_lsn:{lsn:13286744064}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1743618797082332550, v:0}, all_id_meta:{id_meta:[{limited_id:1743655279117594735, latest_log_ts:{val:1743655262469466896, v:0}}, {limited_id:49000001, latest_log_ts:{val:1743605935907428857, v:0}}, {limited_id:360000001, latest_log_ts:{val:1743640357777174079, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:159, palf_env_:0x2b730a3fc030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:7936, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:20.680058] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [17830][T1_Flush][T1][Y0-0000000000000000-0-0] [lt=231] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-04-03 14:01:20.681055] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.681082] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=22] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.688477] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.692069] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.692157] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4683106660, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:20.695513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B7C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.696512] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB5-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.696633] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB5-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=7470, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.696783] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28814 [2025-04-03 14:01:20.702471] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.706468] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=64] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.708358] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB6-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.709158] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.714199] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.714221] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.714234] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.714243] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.724368] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.731037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B81-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.732821] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.732840] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28822 [2025-04-03 14:01:20.732871] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28822 [2025-04-03 14:01:20.732892] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.735707] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=3] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660080731856]}) [2025-04-03 14:01:20.737381] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660080735777]}) [2025-04-03 14:01:20.738416] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660080737412]}) [2025-04-03 14:01:20.739870] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660080738460]}) [2025-04-03 14:01:20.740407] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB9-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.740455] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DB9-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=548, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.741524] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660080740174]}) [2025-04-03 14:01:20.750071] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=39] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.754878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B83-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.761773] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:313) [17546][Blacklist][T0][Y0-0000000000000000-0-0] [lt=29] blacklist_loop exec finished(cost_time=21, is_enabled=true, send_cnt=0) [2025-04-03 14:01:20.762096] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.762112] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28824 [2025-04-03 14:01:20.762277] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28824 [2025-04-03 14:01:20.762286] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.762297] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBB-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.762367] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBB-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=859, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.766460] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.766709] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.768834] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4815682987, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.768872] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.770691] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [17796][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=100] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:812, free_obj_hold_cnt:138, allocator used:3777200, allocator total:3916800}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:120, free_obj_hold_cnt:2, allocator used:8002102, allocator total:8002102}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=11918902, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:30, free_obj_hold_cnt:19, allocator used:97216, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=932) [2025-04-03 14:01:20.772301] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.773199] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBC-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.773248] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBC-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=678, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.781177] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:20.781203] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28826 [2025-04-03 14:01:20.781249] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28826 [2025-04-03 14:01:20.781266] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.785160] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.786563] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBD-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.786623] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBD-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2948, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.794761] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4683208859, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:20.794799] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.797946] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.798030] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=76] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28828 [2025-04-03 14:01:20.798140] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28828 [2025-04-03 14:01:20.798189] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.799254] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBE-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.799305] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBE-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2338, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.810187] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBF-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.810231] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DBF-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=299, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.812563] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=60] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:20.812609] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=39] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28830 [2025-04-03 14:01:20.812665] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28830 [2025-04-03 14:01:20.812674] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:20.815218] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.815241] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.815250] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.815259] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.824787] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.824992] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC0-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.825040] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC0-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=3121, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.831446] INFO eloop_run (eloop.c:127) [17464][pnio2][T0][Y0-0000000000000000-0-0] [lt=41] PNIO [ratelimit] time: 1743660080831442, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007598, add_bytes: 0 [2025-04-03 14:01:20.839307] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC1-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.839348] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC1-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=3018, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.843121] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660080842320]}) [2025-04-03 14:01:20.844297] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660080843170]}) [2025-04-03 14:01:20.845158] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660080844314]}) [2025-04-03 14:01:20.847047] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660080845178]}) [2025-04-03 14:01:20.847928] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:676) [17564][MysqlIO][T0][Y0-0000000000000000-0-0] [lt=35] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:20.849860] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17560][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:20.849866] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660080847511]}) [2025-04-03 14:01:20.850343] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC2-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.850395] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC2-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=894, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.851042] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17534][DiskUseReport][T1001][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.851255] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17562][BatchIO][T0][Y0-0000000000000000-0-0] [lt=12] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:20.851535] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:641) [17561][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:20.852032] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.852127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.852396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.852573] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.852654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.852755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.853154] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:603) [17569][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-04-03 14:01:20.853627] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.853711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.853832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854035] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854696] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854761] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854912] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.854976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855115] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855339] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855619] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855757] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.855970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856047] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856115] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856181] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856315] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856477] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856563] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856630] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.856982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.857055] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.857119] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.857240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.857361] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.857556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.857688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.857882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.858015] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.858210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.858539] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.858673] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.859847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.860916] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.860940] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC3-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.862501] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.862541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.864049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.864084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.865188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.865214] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.865423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.865449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.866609] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=4815781030, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.866647] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=34] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.867955] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.868017] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=60] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.869113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.869139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.869446] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.869472] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.869702] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.869729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B84-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.870546] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:20.871073] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=45] Wash time detail, (compute_wash_size_time=225, refresh_score_time=91, wash_time=15) [2025-04-03 14:01:20.874254] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=24][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.875398] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4273, dropped:16148, tid:17514}]) [2025-04-03 14:01:20.875568] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC4-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.875631] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC4-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=4310, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.876534] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5435, clean_start_pos=1321194, clean_num=62914) [2025-04-03 14:01:20.879604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.879650] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=44] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.879662] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.879670] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.879755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.879768] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.879775] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.879791] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.879874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.879889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.879896] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.879901] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.879967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.879982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.879988] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.879994] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.880071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880096] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=12][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.880101] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.880224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880242] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880249] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.880254] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.880312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880325] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880331] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.880336] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.880455] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880483] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.880489] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.880562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880581] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.880587] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.880645] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880664] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.880694] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=30][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.880762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880775] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880781] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.880786] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.880860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.880880] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.880884] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.881297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881333] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=16][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.881339] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.881407] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881428] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.881433] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.881500] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881521] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.881527] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.881585] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881605] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881613] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.881618] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.881681] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881694] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881700] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.881705] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.881770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881791] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.881796] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.881924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.881950] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.881955] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.882032] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882050] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.882055] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.882256] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882283] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.882299] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.882396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882416] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.882422] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.882507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882518] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882529] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.882539] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.882634] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882644] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882662] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=17][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.882671] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.882747] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882767] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.882776] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.882869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882879] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.882890] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.882899] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.882997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883035] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.883041] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.883127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883133] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883144] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.883154] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.883330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883339] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883349] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.883368] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.883612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883620] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883626] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.883632] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.883729] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883748] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.883754] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.883862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883881] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.883895] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=13][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.883984] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.883995] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884014] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=19][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.884023] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.884162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884170] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884176] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.884181] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.884277] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884287] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884303] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=15][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.884314] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.884453] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884463] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884474] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.884485] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.884576] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884588] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.884594] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.884735] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884745] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884753] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=8][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.884762] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.884872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884888] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.884897] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.884990] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.884997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885013] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=15][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885024] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885118] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885130] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885141] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885147] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885231] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885241] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885251] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885259] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=8][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885359] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885370] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885390] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=19][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885396] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885344] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.885492] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885514] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885519] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885618] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885628] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885634] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885649] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=15][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885733] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885738] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885804] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885823] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885831] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=7][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885836] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885909] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.885915] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.885921] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.885997] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886012] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886017] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.886021] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.886085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886095] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.886099] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.886161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886165] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886170] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.886180] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.886243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886248] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886253] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.886257] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.886365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886377] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.886382] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.886445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886451] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886462] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=11][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.886466] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.886532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886537] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886542] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.886547] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.886686] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886703] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.886708] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=4][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.886975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886988] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.886994] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887010] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.887088] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887095] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.887235] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887240] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.887356] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887366] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=10][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887437] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887443] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887516] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887521] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887591] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887596] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887659] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.887666] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887671] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887735] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887752] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=16][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.887839] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887845] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887904] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887909] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.887973] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.887980] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=1][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.887985] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=5][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.888054] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.888066] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=12][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.888135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.888142] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.888148] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.888209] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0][errcode=-4273] sync_wash_mbs failed(ret=-4273, tenant_id=1003, wash_size=6291456) [2025-04-03 14:01:20.888215] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:112) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=6][errcode=-4273] after wash from cache, still can't alloc chunk from chunk_mgr, maybe alloc by other thread(size=2080768, wash_size=6291456, ret=-4273) [2025-04-03 14:01:20.888332] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.888507] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.888665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.888785] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.889221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.889236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.889316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.889509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.889611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.889800] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B85-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.893402] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=46] get wrs ts(ls_id={id:1}, delta=4683307027, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:20.893449] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.898341] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC6-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.898410] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC6-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=323, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.904219] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=2] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=4) [2025-04-03 14:01:20.909018] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC7-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.909079] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC7-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=536, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.915289] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.915318] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=24][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.915330] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.915363] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.917130] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [18417][T1003_Flush][T1003][Y0-0000000000000000-0-0] [lt=1117] ====== traversal_flush timer task ====== [2025-04-03 14:01:20.917286] INFO [MDS] flush (mds_table_mgr.cpp:172) [18417][T1003_Flush][T1003][Y0-0000000000000000-0-0] [lt=51] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b732e1ea510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1003, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743640313064858146, v:0}, clog_base_lsn:{lsn:22815621120}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1743640313064858145, v:0}, all_id_meta:{id_meta:[{limited_id:1743655280713438675, latest_log_ts:{val:1743655264050701910, v:0}}, {limited_id:77000001, latest_log_ts:{val:1743599198385875085, v:0}}, {limited_id:340000001, latest_log_ts:{val:1743648232469086069, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:620, palf_env_:0x2b734b5f8030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:2100442928, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:20.917417] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [18417][T1003_Flush][T1003][Y0-0000000000000000-0-0] [lt=122] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-04-03 14:01:20.919757] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC8-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.919832] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC8-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=570, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.924879] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=71][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:20.930505] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC9-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.930558] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DC9-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=576, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.941611] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCA-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.941687] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCA-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=910, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.945115] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1761) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11][errcode=-4283] gts not ready(ret=-4283, retry_times=2002) [2025-04-03 14:01:20.945175] WDIAG [STORAGE.TRANS] acquire_global_snapshot__ (ob_trans_service_v4.cpp:1783) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=54][errcode=-4283] acquire global snapshot fail(ret=-4283, gts_ahead=0, expire_ts=1743660083969460, now=[mts=1743660079721694], now0=[mts=1743660079721694], snapshot={val:18446744073709551615, v:3}, uncertain_bound=0) [2025-04-03 14:01:20.945226] WDIAG [STORAGE.TRANS] get_read_snapshot (ob_tx_api.cpp:628) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=46][errcode=-4283] acquire global snapshot fail(ret=-4283, tx={this:0x2b7399219100, tx_id:{txid:47719472}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660079720402, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:20.945380] WDIAG [SQL.EXE] stmt_setup_snapshot_ (ob_sql_trans_control.cpp:745) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=137][errcode=-4283] fail to get snapshot(ret=-4283, local_single_ls_plan=false, first_ls_id={id:-1}, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}) [2025-04-03 14:01:20.945469] WDIAG [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:559) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=86][errcode=-4283] fail to exec stmt_setup_snapshot_(session, das_ctx, plan, plan_ctx, txs)(ret=-4283, session_id=1, *tx_desc={this:0x2b7399219100, tx_id:{txid:47719472}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660079720402, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}) [2025-04-03 14:01:20.945598] INFO [SQL.EXE] start_stmt (ob_sql_trans_control.cpp:627) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=122] start stmt(ret=-4283, auto_commit=true, session_id=1, snapshot={this:0x2b72f7dce008, valid:false, source:0, core:{version:{val:18446744073709551615, v:3}, tx_id:{txid:0}, scn:0}, uncertain_bound:0, snapshot_lsid:{id:-1}, snapshot_ls_role:0, parts:[], committed:false}, savepoint=1743660079720403, tx_desc={this:0x2b7399219100, tx_id:{txid:47719472}, state:1, addr:"172.20.0.5:2882", tenant_id:1, session_id:1, assoc_session_id:1, xid:NULL, xa_mode:"", xa_start_addr:"0.0.0.0:0", access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:0, op_sn:2, alloc_ts:1743660079720402, active_ts:-1, commit_ts:-1, finish_ts:-1, timeout_us:29999990, lock_timeout_us:-1, expire_ts:9223372036854775807, coord_id:{id:-1}, parts:[], exec_info_reap_ts:0, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:17180000522, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, can_elr:false, cflict_txs:[], abort_cause:0, commit_expire_ts:-1, commit_task_.is_registered():false, modified_tables:[], ref:2}, plan_type=4, stmt_type=2, has_for_update=false, query_start_time=1743660054039470, use_das=false, nested_level=0, session={this:0x2b73937661a0, id:1, deser:false, tenant:"sys", tenant_id:1, effective_tenant:"sys", effective_tenant_id:1, database:"oceanbase", user:"root@%", consistency_level:3, session_state:0, autocommit:true, tx:0x2b7399219100}, plan=0x2b73976ba050, consistency_level_in_plan_ctx=3, trans_result={incomplete:false, parts:[], touched_ls_list:[], cflict_txs:[]}) [2025-04-03 14:01:20.945758] WDIAG [SQL] start_stmt (ob_result_set.cpp:351) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=158][errcode=-4283] fail to start stmt(ret=-4283, phy_plan->get_dependency_table()=[{table_id:342, schema_version:1737450199263424, object_type:1, is_db_explicit:false, is_existed:true}]) [2025-04-03 14:01:20.945846] WDIAG [SQL] do_open_plan (ob_result_set.cpp:536) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=84][errcode=-4283] fail start stmt(ret=-4283) [2025-04-03 14:01:20.945856] WDIAG [SQL] open (ob_result_set.cpp:158) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=-4283] execute plan failed(ret=-4283) [2025-04-03 14:01:20.945865] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=6][errcode=-4283] open result set failed(ret=-4283) [2025-04-03 14:01:20.945886] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15][errcode=-4283] result set open failed(ret=-4283, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}) [2025-04-03 14:01:20.945931] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=43][errcode=-4283] execute failed(ret=-4283, tenant_id=1, executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, retry_cnt=21, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:20.945969] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:91) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=23] will sleep(sleep_us=21000, remain_us=3093495, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=21, v.err_=-4283, timeout_timestamp=1743660084039460) [2025-04-03 14:01:20.952303] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660080950878]}) [2025-04-03 14:01:20.952562] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCB-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.952621] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCB-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=765, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.952703] WARN alloc_chunk (achunk_mgr.cpp:326) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4013] No memory or reach tenant memory limit(msg="oops, over total memory limit", hold=10735362048, limit=10737418240) [2025-04-03 14:01:20.952902] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660080952324]}) [2025-04-03 14:01:20.953543] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660080952910]}) [2025-04-03 14:01:20.954187] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660080953561]}) [2025-04-03 14:01:20.954786] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660080954369]}) [2025-04-03 14:01:20.955684] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.955715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.955849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.955889] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956014] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956545] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956746] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956880] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956888] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.956977] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.957282] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.957304] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.957629] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.957653] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.957849] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.957869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.958365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.958399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.958728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.958751] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.958897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.958918] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959035] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959211] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959372] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959466] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959502] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959915] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.959935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960272] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960288] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960435] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960594] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960612] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960854] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.960882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.961033] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.961041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.961129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.961145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.961531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.961551] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.961754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.961794] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.962289] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.962309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.962528] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.962548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.962690] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.962715] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.962856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.962873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963013] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963037] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963179] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963301] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963427] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963542] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963667] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.963682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.965109] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCC-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.965157] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCC-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1320, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.965382] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.965401] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.965531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.965550] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.965674] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.965689] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.965848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.965868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966144] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966160] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966333] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966456] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966471] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966601] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966607] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966701] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966941] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.966945] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967063] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967113] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967118] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967129] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=58][errcode=-4283] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:21, local_retry_times:21, err_:-4283, err_:"OB_GTS_NOT_READY", retry_type:1, client_ret:-4283}, need_retry=true, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:20.967199] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=51][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:20.967208] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=-4283] result set close failed(ret=-4283) [2025-04-03 14:01:20.967215] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=6][errcode=-4283] failed to close result(close_ret=-4283, ret=-4283) [2025-04-03 14:01:20.967242] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967236] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8][errcode=-4283] failed to process record(executor={ObIExecutor:, sql:"INSERT INTO __all_ls_meta_table (tenant_id, ls_id, svr_ip, svr_port, sql_port, role, member_list, proposal_id, replica_type, replica_status, restore_status, memstore_percent, unit_id, zone, paxos_replica_number, data_size, required_size, rebuild, learner_list) VALUES (1, 1, '172.20.0.5', 2882, 2881, 2, '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', 0, 0, 'NORMAL', 0, 100, 3, 'zone3', 3, 0, 0, 0, '') ON DUPLICATE KEY UPDATE sql_port = 2881, role = 2, member_list = '172.20.0.3:2882:1,172.20.0.4:2882:1,172.20.0.5:2882:1743648235302292', proposal_id = 0, replica_type = 0, replica_status = 'NORMAL', restore_status = 0, memstore_percent = 100, unit_id = 3, zone = 'zone3', paxos_replica_number = 3, data_size = 0, required_size = 0, rebuild = 0, learner_list = ''"}, record_ret=-4283, ret=-4283) [2025-04-03 14:01:20.967246] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967296] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967299] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967495] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:20.967524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967741] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967754] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967820] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967797] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=32] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967952] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.967957] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968044] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968087] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968107] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968114] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968207] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968213] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968226] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968401] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968512] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968608] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968610] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4815882908, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:20.968615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968623] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.968711] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968932] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.968949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969056] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=14] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969200] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969460] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969476] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969546] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969679] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969705] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.969914] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970031] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970099] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970166] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970171] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970230] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970309] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970371] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970415] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.970374] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18253][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=43] start do ls ha handler(ls_id_array_=[{id:1002}, {id:1001}, {id:1}]) [2025-04-03 14:01:20.970448] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:251) [17485][LSMetaTblUp0][T1][YB42AC140005-000631D6BABCBF59-0-0] [lt=1][errcode=-4023] query gts fail(tmp_ret=-4013, leader="172.20.0.4:2882") [2025-04-03 14:01:20.970491] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970592] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970822] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970832] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.970898] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.971153] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.971410] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.971739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.971933] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.972253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.972350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.972536] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.972680] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.973052] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.973367] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B86-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.974344] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=45][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.974648] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.974666] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.974738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.974868] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.975497] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.975526] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=28] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.975563] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4013, dropped:4246, tid:17533}]) [2025-04-03 14:01:20.975657] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.976249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.976397] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.976506] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.976523] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=15][errcode=-4013] oops, alloc failed, tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=9223372036854775807, tenant_hold=8096894976, tenant_limit=9223372036854775807 [2025-04-03 14:01:20.976535] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=8][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:20.976543] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=8][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:20.976551] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=8][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:20.976558] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=7][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:20.976568] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=6][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:20.976605] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=37] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.976627] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=21][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.976659] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=31] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1367, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.976675] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCD-0-0] [lt=15][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.976867] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.976942] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.977117] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.977283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.977603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.977784] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.977971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.978092] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.978224] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:20.978238] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=14][errcode=-4013] rpc memory pool alloc memory failed(sz=1374, alloc_sz=16256) [2025-04-03 14:01:20.978245] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=6][errcode=-4013] alloc buffer fail(payload_sz=1190) [2025-04-03 14:01:20.978250] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=5][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:20.978265] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=10][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1003, svr="172.20.0.3:2882", timeout=997668, timeout_timestamp=1743660081973688) [2025-04-03 14:01:20.978287] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=21][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:20.978295] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=6][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:20.978302] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=6][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:20.978323] WDIAG [SQL] open (ob_result_set.cpp:158) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=20][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:20.978329] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=5][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:20.978334] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=4][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:20.978347] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=12][errcode=-4013] execute failed(ret=-4013, tenant_id=1003, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737618065792656) [2025-04-03 14:01:20.978362] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=13][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:20.978385] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=17][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:20.978394] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=8][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:20.978399] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=4][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:20.978521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.978534] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.978842] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.978927] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.979085] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.979243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.979391] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.979615] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.979723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.979924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.980069] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.980199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.980369] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.980695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.980850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.981016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.981190] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.981355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.981503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.981672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.981835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.982010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.982194] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.982347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.982488] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1003, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:20.982499] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B87-0-0] [lt=10][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:20.982506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=6][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:20.982512] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=5][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:20.982517] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=5][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1003) [2025-04-03 14:01:20.982522] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=5][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1003) [2025-04-03 14:01:20.982527] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:382) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=4][errcode=-4013] query failed(ret=-4013, conn=0x2b7376644050, start=1743660080973696, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port, cluster_id=1) [2025-04-03 14:01:20.982539] WDIAG [SHARE.PT] batch_get (ob_persistent_ls_table.cpp:864) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=10][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", exec_tenant_id=1003, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1004 AND ls_id IN (1) ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:20.982580] WDIAG [SHARE.PT] batch_get (ob_ls_table_operator.cpp:399) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=7][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1004) [2025-04-03 14:01:20.982595] WDIAG [SHARE.LOCATION] batch_renew_ls_locations (ob_ls_location_service.cpp:1250) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=14][errcode=-4013] fail to batch get ls info by operator(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:20.982605] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1005) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=9][errcode=-4013] batch renew ls locations failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_id=1, tenant_id=1004, ls_ids=[{id:1}]) [2025-04-03 14:01:20.982610] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:560) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=5][errcode=-4013] fail to renew location(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", task={cluster_id:1, tenant_id:1004, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1743660080954781}) [2025-04-03 14:01:20.982618] WDIAG [SERVER] batch_process_tasks (ob_uniq_task_queue.h:503) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=7][errcode=-4013] fail to batch process task(ret=-4013) [2025-04-03 14:01:20.982621] WDIAG [SERVER] run1 (ob_uniq_task_queue.h:454) [17514][UserLocAsyncUp0][T0][YB42AC140005-000631D6BD3F2B87-0-0] [lt=3][errcode=-4013] fail to batch execute task(ret=-4013, tasks.count()=1) [2025-04-03 14:01:20.985404] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=97][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:20.987131] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=17][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:20.987148] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=17][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:20.987158] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=10][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:20.987163] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:20.987168] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=3][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:20.987195] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=27] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.987206] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=11][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.987217] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=10] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=440, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.987227] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCE-0-0] [lt=9][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.993553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=5] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.993595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=39] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.993716] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.993739] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.993866] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.993900] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994267] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994294] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994426] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994527] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994551] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994654] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994664] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994721] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994838] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994926] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.994939] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995041] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995164] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995186] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995297] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:20.995654] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_WRS_CLUSTER_HEARTBEAT, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:20.995684] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:20.995710] WDIAG [STORAGE.TRANS] post_cluster_heartbeat_rpc_ (ob_tenant_weak_read_service.cpp:911) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4013] post cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", cluster_service_master="172.20.0.3:2882", tenant_id_=1001, req={req_server:"172.20.0.5:2882", version:{val:1743659476426054000, v:0}, valid_part_count:0, total_part_count:1, generate_timestamp:1743660080993431}, cluster_service_tablet_id={id:226}) [2025-04-03 14:01:20.995758] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:451) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=45][errcode=-4013] tenant weak read service cluster heartbeat RPC fail(ret=-4013, rcode={code:-4013, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="172.20.0.5:2882", cluster_service_tablet_id={id:226}) [2025-04-03 14:01:20.995832] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:867) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=72][errcode=-4013] post cluster heartbeat rpc fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, local_server_version={val:1743659476426054000, v:0}, valid_part_count=0, total_part_count=1, generate_timestamp=1743660080993431) [2025-04-03 14:01:20.995864] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:877) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4013] tenant weak read service do cluster heartbeat fail(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1743660080794715, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="172.20.0.3:2882") [2025-04-03 14:01:20.995964] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=4683409564, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:20.996034] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:20.998949] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=10][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:20.999014] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=47][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:20.999035] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=20][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:20.999055] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=18][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:20.999084] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=18][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:20.999148] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=63] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.999191] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=41][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.999233] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=41] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1753, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:20.999284] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DCF-0-0] [lt=48][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:20.999750] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:20.999833] WDIAG common_alloc (ob_tenant_ctx_allocator.cpp:469) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=80][errcode=-4013] oops, alloc failed, tenant_id=500, ctx_id=16, ctx_name=RPC_CTX_ID, ctx_hold=0, ctx_limit=9223372036854775807, tenant_hold=8096894976, tenant_limit=9223372036854775807 [2025-04-03 14:01:20.999890] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=46][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:20.999913] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:20.999940] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=23][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, arg={addr:"172.20.0.4:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.000025] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=83][errcode=-4013] call rpc func failed(server="172.20.0.4:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.4:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.000069] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:83) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=40][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.005485] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.005525] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=39][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:21.005541] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=11][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, arg={addr:"172.20.0.3:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.005562] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=20][errcode=-4013] call rpc func failed(server="172.20.0.3:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.3:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.006089] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_DETECT_MASTER_RS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.006115] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=25][errcode=-4013] rpc memory pool alloc memory failed(sz=24, alloc_sz=16256) [2025-04-03 14:01:21.006135] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:356) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=18][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, arg={addr:"172.20.0.5:2882", cluster_id:1}, cluster_id=1, tenant_id=1, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.006159] WDIAG [RPC] call (ob_async_rpc_proxy.h:290) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=23][errcode=-4013] call rpc func failed(server="172.20.0.5:2882", timeout=2000000, cluster_id=1, tenant_id=1, arg={addr:"172.20.0.5:2882", cluster_id:1}, group_id=9, ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.006220] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:21.006250] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:847) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=28][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-04-03 14:01:21.006261] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:140) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-04-03 14:01:21.006284] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:151) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-04-03 14:01:21.006307] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:944) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743659803760242, replica_locations:[{server:"172.20.0.3:2882", role:2, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:0}, {server:"172.20.0.4:2882", role:1, sql_port:2881, replica_type:0, property:{memstore_percent_:100}, restore_status:{status:0}, proposal_id:159}]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1}, renew_time:1743660081006305, replica_locations:[]}) [2025-04-03 14:01:21.007413] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007455] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=40] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007547] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007792] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=35] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007891] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.007903] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008274] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=19] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=25] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008515] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008599] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008617] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008883] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.008901] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.009872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.009894] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.009982] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=38] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010091] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010149] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [17844][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-04-03 14:01:21.010177] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010341] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010366] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=24] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010436] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.010553] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD0-0-0] [lt=40][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.010576] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD0-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:21.010585] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.010599] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD0-0-0] [lt=22][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:21.010602] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=16][errcode=-4013] rpc memory pool alloc memory failed(sz=1333, alloc_sz=16256) [2025-04-03 14:01:21.010608] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD0-0-0] [lt=8][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:21.010612] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] alloc buffer fail(payload_sz=1149) [2025-04-03 14:01:21.010631] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=17][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:21.010649] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1, svr="172.20.0.4:2882", timeout=998150, timeout_timestamp=1743660082006342) [2025-04-03 14:01:21.010620] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD0-0-0] [lt=7][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.010672] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=21][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:21.010687] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=11][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:21.010675] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD0-0-0] [lt=54] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.010700] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=10][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:21.010711] WDIAG [SQL] open (ob_result_set.cpp:158) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:21.010702] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD0-0-0] [lt=25][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.010724] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:21.010730] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD0-0-0] [lt=2][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.010735] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:21.010762] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=26][errcode=-4013] execute failed(ret=-4013, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1743590121028944) [2025-04-03 14:01:21.010780] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=15][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:21.010851] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=56][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:21.010863] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=11][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:21.010871] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=8][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:21.010899] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:21.010914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=12][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:21.010927] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=10][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:21.010940] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=12][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1) [2025-04-03 14:01:21.010951] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=10][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1) [2025-04-03 14:01:21.010967] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=14][errcode=-4013] query failed(ret=-4013, conn=0x2b738ffa6050, start=1743660081007314, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:21.010981] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=14][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:21.010993] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:21.011096] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=28][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001) [2025-04-03 14:01:21.011111] WDIAG [SHARE.LOCATION] renew_location_for_tenant (ob_ls_location_service.cpp:806) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=13][errcode=-4013] fail to get all ls info(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001, ls_infos=[]) [2025-04-03 14:01:21.011139] WDIAG [SHARE.LOCATION] renew_all_ls_locations (ob_ls_location_service.cpp:737) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=26][errcode=-4013] renew cache for tenant failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1001) [2025-04-03 14:01:21.011956] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.011985] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.012175] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.012549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.012621] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.012794] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.012949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.013145] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.013283] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.013462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.013626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.013670] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.013682] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=12][errcode=-4013] rpc memory pool alloc memory failed(sz=1301, alloc_sz=16256) [2025-04-03 14:01:21.013690] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=7][errcode=-4013] alloc buffer fail(payload_sz=1117) [2025-04-03 14:01:21.013697] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=6][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:21.013708] WDIAG [SQL.EXE] task_execute_v2 (ob_executor_rpc_impl.cpp:143) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=6][errcode=-4013] rpc task_execute fail(ret=-4013, tenant_id=1001, svr="172.20.0.3:2882", timeout=998647, timeout_timestamp=1743660082011148) [2025-04-03 14:01:21.013719] WDIAG [SQL.EXE] execute_with_sql (ob_remote_scheduler.cpp:252) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=10][errcode=-4013] task execute failed(ret=-4013) [2025-04-03 14:01:21.013730] WDIAG [SQL.EXE] schedule (ob_remote_scheduler.cpp:53) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=10][errcode=-4013] execute with sql failed(ret=-4013) [2025-04-03 14:01:21.013737] WDIAG [SQL] do_open_plan (ob_result_set.cpp:546) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=6][errcode=-4013] fail execute plan(ret=-4013) [2025-04-03 14:01:21.013743] WDIAG [SQL] open (ob_result_set.cpp:158) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=5][errcode=-4013] execute plan failed(ret=-4013) [2025-04-03 14:01:21.013749] WDIAG [SERVER] open (ob_inner_sql_result.cpp:153) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=4][errcode=-4013] open result set failed(ret=-4013) [2025-04-03 14:01:21.013754] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:696) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=5][errcode=-4013] result set open failed(ret=-4013, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}) [2025-04-03 14:01:21.013764] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:831) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] execute failed(ret=-4013, tenant_id=1001, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1743590121028944, local_tenant_schema_version=1737452026425991) [2025-04-03 14:01:21.013775] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:955) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-4013, err_:"OB_ALLOCATE_MEMORY_FAILED", retry_type:0, client_ret:-4013}, need_retry=false, THIS_WORKER.can_retry()=false, v.ctx_.multi_stmt_item_={is_part_of_multi_stmt:false, seq_num:0, sql:"", batched_queries:NULL, is_ps_mode:false, ab_cnt:0}) [2025-04-03 14:01:21.013795] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:218) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=15][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:21.013802] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:198) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=6][errcode=-4013] result set close failed(ret=-4013) [2025-04-03 14:01:21.013807] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:836) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=5][errcode=-4013] failed to close result(close_ret=-4013, ret=-4013) [2025-04-03 14:01:21.013852] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.013938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.014098] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.014262] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.014600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.014764] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.014906] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.015080] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.015234] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.015368] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.015393] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.015407] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.015419] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.015552] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.015905] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.016050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.016208] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.016375] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.016538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.016698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.016805] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1001, label=MysqlRequesReco, ctx_id=0, prio=0, nbyte=2079744) [2025-04-03 14:01:21.016831] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:866) [17518][AutoLSLoc][T1001][YB42AC140005-000631D6C41CBE7B-0-0] [lt=24][errcode=-4013] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-4013, ret=-4013) [2025-04-03 14:01:21.016842] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:892) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-4013, ret=-4013) [2025-04-03 14:01:21.016851] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1647) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=7][errcode=-4013] execute sql failed(ret=-4013, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:21.016858] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:949) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=6][errcode=-4013] retry_while_no_tenant_resource failed(ret=-4013, tenant_id=1001) [2025-04-03 14:01:21.016864] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1587) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=5][errcode=-4013] execute_read failed(ret=-4013, cluster_id=1, tenant_id=1001) [2025-04-03 14:01:21.016871] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:131) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=6][errcode=-4013] query failed(ret=-4013, conn=0x2b73ba17a050, start=1743660081011889, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:21.016880] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:66) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=8][errcode=-4013] read failed(ret=-4013) [2025-04-03 14:01:21.016892] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=10][errcode=-4013] execute sql failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-04-03 14:01:21.016941] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=8][errcode=-4013] get all ls info by persistent_ls_ failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:21.016949] WDIAG [SHARE.LOCATION] renew_location_for_tenant (ob_ls_location_service.cpp:806) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=7][errcode=-4013] fail to get all ls info(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002, ls_infos=[]) [2025-04-03 14:01:21.016958] WDIAG [SHARE.LOCATION] renew_all_ls_locations (ob_ls_location_service.cpp:737) [17518][AutoLSLoc][T0][YB42AC140005-000631D6C41CBE7B-0-0] [lt=9][errcode=-4013] renew cache for tenant failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1002) [2025-04-03 14:01:21.017524] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.017676] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.017871] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.018090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.018161] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.018365] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.018496] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.018658] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.018827] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.018974] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.019106] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17518][AutoLSLoc][T1][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0][errcode=-4013] allocate memory fail(attr=tenant_id=1, label=OB_REMOTE_SYNC_EXECUTE, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.019756] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.019971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.020045] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.020210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.020532] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.020698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.021040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.021199] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.021335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.021502] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.021551] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD1-0-0] [lt=26][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.021574] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD1-0-0] [lt=22][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:21.021582] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD1-0-0] [lt=7][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:21.021589] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD1-0-0] [lt=6][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:21.021618] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD1-0-0] [lt=24][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.021651] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD1-0-0] [lt=1][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.021677] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD1-0-0] [lt=25] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=786, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.021698] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD1-0-0] [lt=20][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.022028] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.022150] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.022316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.022668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.022847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.022976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.023146] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.023290] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.023473] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.023637] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17518][AutoLSLoc][T1003][YB42AC140005-000631D6C41CBE7B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.024977] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:21.032415] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=18][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.032436] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=21][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:21.032442] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=5][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:21.032447] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:21.032453] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=3][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.032485] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=32] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.032501] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=16][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.032519] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=18] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=653, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.032537] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:345) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD2-0-0] [lt=18][errcode=-4013] fail to update replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", tenant_id=1, ls_id={id:1}, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.042931] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD3-0-0] [lt=13][errcode=-4013] allocate memory fail(attr=tenant_id=500, label=OB_REPORT_SYS_LS, ctx_id=16, prio=0, nbyte=16256) [2025-04-03 14:01:21.042946] WDIAG [RPC.OBRPC] rpc_mem_pool_create_page (ob_rpc_mem_pool.cpp:56) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD3-0-0] [lt=14][errcode=-4013] rpc memory pool alloc memory failed(sz=342, alloc_sz=16256) [2025-04-03 14:01:21.042951] WDIAG [RPC.OBRPC] rpc_encode_req (ob_rpc_endec.h:65) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD3-0-0] [lt=5][errcode=-4013] alloc buffer fail(payload_sz=158) [2025-04-03 14:01:21.042956] WDIAG [RPC] send (ob_poc_rpc_proxy.h:147) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD3-0-0] [lt=4][errcode=-4013] rpc encode req fail(ret=-4013) [2025-04-03 14:01:21.042961] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:193) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD3-0-0] [lt=3][errcode=-4013] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.20.0.4:2882", ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED") [2025-04-03 14:01:21.042984] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD3-0-0] [lt=22] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.043021] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD3-0-0] [lt=36][errcode=-4013] update replica failed(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.043034] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD3-0-0] [lt=12] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=397, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.053414] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD4-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.053444] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD4-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=308, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.056102] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660081055161]}) [2025-04-03 14:01:21.058437] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660081056134]}) [2025-04-03 14:01:21.059242] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660081058461]}) [2025-04-03 14:01:21.059260] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=254, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-04-03 14:01:21.060205] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660081059272]}) [2025-04-03 14:01:21.060223] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:70) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] gts statistics(tenant_id=1003, gts_rpc_cnt=0, get_gts_cache_cnt=150, get_gts_with_stc_cnt=8334, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-04-03 14:01:21.060699] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.060714] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.060720] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.060848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.060853] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.060961] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.060967] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061127] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660081060422]}) [2025-04-03 14:01:21.061240] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061249] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061347] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061449] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061538] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061543] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061629] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061726] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061733] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061976] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.061980] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062131] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062138] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062182] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062187] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062279] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062395] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062405] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062504] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062561] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062798] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062804] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062954] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.062966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063043] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063050] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063104] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063109] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063201] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063206] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063305] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063417] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063423] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063510] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063553] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063677] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063765] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063869] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063949] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.063965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064038] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064123] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064191] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064195] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064266] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064322] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064311] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD5-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.064351] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD5-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=786, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.064385] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064389] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064482] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064488] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064590] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064638] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064713] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064718] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064822] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064828] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064881] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064886] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064970] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.064975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065095] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065100] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065225] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065311] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065357] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065362] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065418] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065489] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065498] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065544] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065604] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065611] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065665] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065670] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065723] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065728] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065780] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065850] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065855] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065959] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.065966] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.066025] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.066029] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.066222] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.066350] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.066533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.066685] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.066824] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.067008] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.067530] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.067668] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.067882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.068016] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.068316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B88-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.069306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.069485] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.069819] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.070081] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.070444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.070736] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.071057] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.073024] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.073329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.073531] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.073649] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.073874] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.073971] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.074198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.074400] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=44][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.074633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.074965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.075130] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.075331] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.075516] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.075669] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD6-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.075718] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.075782] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.075790] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=7] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.075797] WDIAG [SHARE] refresh (ob_task_define.cpp:402) [17422][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=13][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:8405, tid:17422}]) [2025-04-03 14:01:21.075916] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076116] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076124] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=7] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.076165] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076205] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076263] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076351] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076399] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076408] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.076456] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076505] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076560] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.076615] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.077008] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:21.080882] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash time detail, (compute_wash_size_time=136, refresh_score_time=46, wash_time=6) [2025-04-03 14:01:21.082910] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=2012, clean_start_pos=1384108, clean_num=62914) [2025-04-03 14:01:21.083181] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.083205] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083210] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083252] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.083275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083278] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083368] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.083394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=25] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083402] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083488] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.083505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083574] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.083591] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083595] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083685] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.083697] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083762] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.083926] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.083938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.083976] WDIAG [SERVER] record_request (ob_mysql_request_manager.cpp:177) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B89-0-0] [lt=1][errcode=0] record concurrent fifoallocator alloc mem failed(total_size=1605, tenant_id_=1003, mem_limit_=107374182, request_id_=0, ret=0) [2025-04-03 14:01:21.084682] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.084703] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=20] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.084710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.084805] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.084835] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=29] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.084879] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.084930] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085024] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085039] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.085092] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085181] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085255] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085299] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085314] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.085390] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085457] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085472] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.085547] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085861] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.085878] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.085987] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.086007] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.086131] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.086201] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.086256] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.086269] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.086336] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.086440] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090048] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 3 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090067] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090074] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090121] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090132] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090135] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090225] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090243] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090247] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090312] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090326] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090330] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=3] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090367] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090373] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=5] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090410] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090481] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090589] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090597] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=7] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090692] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090896] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.090907] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.090979] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091090] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091250] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.091311] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091371] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091418] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091467] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091480] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.091515] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091560] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091607] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091699] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091765] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091819] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091829] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=9] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.091885] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091942] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.091996] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092063] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092112] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092129] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.092196] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092278] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092387] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092505] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=15] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.092601] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092666] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092841] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092856] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.092921] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.092971] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093197] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093253] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093307] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093353] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093438] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093450] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.093572] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093631] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093679] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093729] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093772] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093786] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.093915] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.093993] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094022] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18702][LeaseHB][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.094147] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094206] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094308] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094320] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.094411] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094474] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094576] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.094698] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094762] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094827] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094890] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.094902] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=12] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.094950] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095010] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095092] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095155] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095229] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095246] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.095361] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095460] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095559] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=11] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.095641] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095732] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095886] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.095951] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.096082] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.096096] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.096229] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.096284] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.096333] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8A-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.097140] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD8-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.097194] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD8-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=753, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.104314] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=3) [2025-04-03 14:01:21.109274] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD9-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.109344] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DD9-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1995, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.115441] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660077109703, last_first_register_time_us=1743648246712872, register_time_us=1743660077109703, curr_time_us=1743660081115439) [2025-04-03 14:01:21.115518] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.115564] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=44] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115574] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115629] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.115646] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115652] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115684] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.115704] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=19] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115710] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115741] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.115755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115760] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115791] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.115825] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115831] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115858] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.115873] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115882] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115913] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.115924] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115929] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115961] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.115984] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.115989] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116035] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116049] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116054] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116097] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116122] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116127] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116159] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116173] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116178] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116208] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116242] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116272] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116295] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116325] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116340] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116345] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116375] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116394] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=18] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116399] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116429] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116443] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116448] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116481] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116499] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116535] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1280 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 43 | get_cnt= 63 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 139 | get_cnt= 328 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 132 | get_cnt= 30605 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 3 | get_cnt= 364 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 319 | get_cnt= 1706 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 584 | get_cnt= 584 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 409 | get_cnt= 1757 | score=0.000000 | [CACHE] tenant_id= 1004 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 18 | get_cnt= 18 | score=0.000000 | [2025-04-03 14:01:21.116549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116554] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.116581] INFO [PALF] check_parent_health (log_config_mgr.cpp:2829) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0] re register_parent success(ret=0, ret="OB_SUCCESS", palf_id=1, self="172.20.0.5:2882") [2025-04-03 14:01:21.116594] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.116612] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.116622] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.116634] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=7][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.119961] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDA-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.120028] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDA-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=515, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.123499] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=18][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.123560] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=56] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.123580] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.123595] WDIAG [COMMON] alloc_chunk (ob_resource_mgr.cpp:86) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=14][errcode=-4273] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.123729] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.123787] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=56] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.123847] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=59] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.124222] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.124263] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=38] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.124276] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.124568] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.124613] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=42] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.124626] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.124747] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.124781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=31] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.124792] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125046] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.125079] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=30] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125090] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125138] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=52][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:21.125209] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.125245] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=34] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125256] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125372] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.125398] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125414] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=16] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125547] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.125571] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=22] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125581] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125724] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.125769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=43] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.125782] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.126045] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.126084] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=36] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.126094] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.126239] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.126265] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=24] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.126275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.126470] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.126505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=32] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.126514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=8] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.127696] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.127762] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=60] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.127781] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=17] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.128099] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.128143] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.128156] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.128314] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.128343] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=27] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.128354] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.128477] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.128521] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=40] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.128533] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.128719] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=1][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.128755] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=33] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.128779] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.129024] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.129062] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=35] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.129075] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.129219] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0][errcode=0] [CACHE] len: 1760 tenant sync wash failed, cache memblock info: [CACHE] tenant_id= 1003 | cache_id= 9 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 115 | get_cnt= 232 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 0 | get_cnt= 0 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 12 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 2 | get_cnt= 1456 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 54 | get_cnt= 76 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 235 | get_cnt= 592 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 15 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 167 | get_cnt= 36346 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 14 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 9 | get_cnt= 436 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 10 | get_cnt= 13 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 6 | get_cnt= 36 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 4 | ref_count= 2 | status= 1 | policy= 1 | kv_cnt= 5 | get_cnt= 41 | score=0.000000 | [CACHE] tenant_id= 1003 | cache_id= 5 | ref_count= 2 | status= 1 | policy= 0 | kv_cnt= 53 | get_cnt= 53 | score=0.000000 | [2025-04-03 14:01:21.129261] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=39] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.129273] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=11] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.129550] INFO [STORAGE] runTimerTask (ob_tenant_tablet_stat_mgr.cpp:1098) [18309][T1003_TabletSta][T1003][Y0-0000000000000000-0-0] [lt=0] TenantTabletStatMgr refresh all tablet stream(MTL_ID()=1003, interval_step=1, last_update_time=1743660081129547) [2025-04-03 14:01:21.131345] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDB-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.131401] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDB-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1231, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.142524] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDC-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.142567] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDC-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1002, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.147840] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.147892] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=47] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.147921] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.152969] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:202) [18023][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=32] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x2b731e9e82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-04-03 14:01:21.153306] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDD-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.153340] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDD-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=665, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.153445] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.153484] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=37] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.154306] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.154337] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=29] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.155705] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.155738] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=31] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.156335] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.156364] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=27] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.159875] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.159897] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=21] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.161169] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.161188] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.161342] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.161352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.161799] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.161834] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=34] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.161964] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.161975] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.162541] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.162557] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.162678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.162688] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.163312] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.163328] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=15] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.163836] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.163860] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=23] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.164383] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDE-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.164469] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDE-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1019, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.164518] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:101) [18108][T1002_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=68] BlockGCTimerTask success(ret=0, cost_time_us=15, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"172.20.0.5:2882", log_dir:"/data/oceanbase_data/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):1536, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):1536, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):315, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-04-03 14:01:21.164529] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.164548] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.164839] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660081161513]}) [2025-04-03 14:01:21.165089] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.165102] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.165344] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.165355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.165672] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.165683] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1002][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1002, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.166829] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.166848] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.166928] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.166935] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.166993] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167011] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167097] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167185] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167196] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167283] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660081164881]}) [2025-04-03 14:01:21.167427] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167433] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.167909] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660081167311]}) [2025-04-03 14:01:21.168053] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168065] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168137] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168142] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168220] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168225] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=4] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168275] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168280] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168342] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168352] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168393] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168403] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168495] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168505] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168565] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168583] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168633] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168643] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.168664] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.168732] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4816082886, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:21.168751] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:21.168876] INFO [STORAGE.TRANS] generate_new_version (ob_tenant_weak_read_server_version_mgr.cpp:120) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] update tenant weak read server version(tenant_id=1, server_version={version:{val:1743659476447168000, v:0}, total_part_count:1, valid_inner_part_count:0, valid_user_part_count:0, epoch_tstamp:1743660081168689}, version_delta=604721708) [2025-04-03 14:01:21.169184] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=2] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.169198] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=13] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.169412] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.169430] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=18] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.169698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.169706] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.170061] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.170071] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=9] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.170227] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.170235] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=7] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.170419] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.170741] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.171462] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8B-0-0] [lt=3] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.171682] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.171862] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.172383] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.172466] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [17796][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=69] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:812, free_obj_hold_cnt:138, allocator used:3777200, allocator total:3916800}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:120, free_obj_hold_cnt:2, allocator used:8002102, allocator total:8002102}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=11918902, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:30, free_obj_hold_cnt:19, allocator used:97216, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=932) [2025-04-03 14:01:21.172513] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.173639] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.174139] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.174458] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.174492] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=46][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.174769] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.175010] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.175606] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.176236] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.176419] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DDF-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.176562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8B-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.177376] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.177938] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.178034] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.178224] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18224][T1002_LSMetaCh][T1001][YB42AC140005-000631D6BE8CE2DC-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.185544] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.188141] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE0-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.189264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B8C-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.198550] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE1-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.203232] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:788) [18497][T1004_PalfGC][T1004][Y0-0000000000000000-0-0] [lt=28] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):9216, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):9216, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):563, sequence:1}) [2025-04-03 14:01:21.209334] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE2-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.210926] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28834 [2025-04-03 14:01:21.211647] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8E-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.211759] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8E-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.214020] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8E-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.215546] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.215607] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=49][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.215640] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.215685] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=38][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.216636] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28836 [2025-04-03 14:01:21.218509] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8F-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.218600] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B8F-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.220215] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE3-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.222498] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154F-0-0] [lt=1][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.225253] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154F-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.225272] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:21.228740] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154F-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.229162] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B90-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.230562] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154F-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.231172] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154F-0-0] [lt=5] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.231194] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE4-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.231219] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE4-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=861, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.232129] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17722][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=26] ====== tenant freeze timer task ====== [2025-04-03 14:01:21.232930] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B91-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.233233] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154F-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.233765] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.236568] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154F-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.236830] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18029][T1001_ReqMemEvi][T1001][YB42AC140005-000631D6C58D154F-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.237171] INFO [COMMON] print_io_status (ob_io_manager.cpp:1415) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=36] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode: read, size: 1.00, iops: 8192.00, rt: 493.00; ) [2025-04-03 14:01:21.237234] INFO [COMMON] print_io_status (ob_io_manager.cpp:1501) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=38] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.237338] INFO [COMMON] print_sender_status (ob_io_struct.cpp:888) [17423][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=27] [IO STATUS SENDER](*this=send_index: 1, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; send_index: 2, req_count: 0, reservation_ts: 9223372036854775807, group_limitation_ts: 9223372036854775807, tenant_limitation_ts: 9223372036854775807, proportion_ts: 9223372036854775807; ) [2025-04-03 14:01:21.238931] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B92-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.242036] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE5-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.247073] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B93-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.250338] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B94-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.250390] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B94-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.251083] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B94-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.252698] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B94-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.252777] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE6-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.263782] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE7-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.263818] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE7-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=800, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.272099] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE300-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.274549] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.274883] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18020][T1001_LSMetaCh][T1][YB42AC140005-000631D6BC9CE300-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.275976] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660081274953]}) [2025-04-03 14:01:21.276690] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660081276010]}) [2025-04-03 14:01:21.278970] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] ====== tenant freeze timer task ====== [2025-04-03 14:01:21.279019] INFO [STORAGE] log_frozen_memstore_info_if_need_ (ob_tenant_freezer.cpp:1555) [17939][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=40] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.279125] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28846 [2025-04-03 14:01:21.279603] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B95-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.283519] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=1] Wash time detail, (compute_wash_size_time=103, refresh_score_time=47, wash_time=7) [2025-04-03 14:01:21.285040] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:21.285626] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.286480] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DE9-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.293444] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B96-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.293614] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4683707429, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:21.293636] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.293773] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B96-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.299453] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.299473] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28852 [2025-04-03 14:01:21.299502] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28852 [2025-04-03 14:01:21.299509] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.304376] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=31457, map_replace_skip_count_=2) [2025-04-03 14:01:21.307373] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DEB-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.307406] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DEB-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=356, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.309487] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.309511] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28854 [2025-04-03 14:01:21.309544] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28854 [2025-04-03 14:01:21.309551] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.315619] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=13][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.315652] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=29][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.315673] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.315684] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=6][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.318275] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DEC-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.318339] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DEC-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=848, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.325554] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:21.325955] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17519][AutoLSLocRpc][T0][YB42AC140005-000631D6C29CE37B-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.329154] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.329173] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28856 [2025-04-03 14:01:21.329210] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28856 [2025-04-03 14:01:21.329221] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.329212] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DED-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.329259] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DED-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=789, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.335318] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.335334] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=13] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28858 [2025-04-03 14:01:21.335364] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28858 [2025-04-03 14:01:21.335370] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.339643] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DEE-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.339672] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DEE-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=279, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.341375] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.341404] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1004][YB42AC140005-000631D6C12CE2E6-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1004, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.342514] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.342588] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.342678] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.342801] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343040] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343116] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343221] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343316] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343396] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343478] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343549] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343622] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343695] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343771] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343872] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.343948] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.345087] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.346058] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.348102] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.349965] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [18605][T1004_LSMetaCh][T1003][YB42AC140005-000631D6C12CE2E6-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.350298] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:38) [18240][T1002_TabletShe][T1002][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.351356] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.353104] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB92A-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.355141] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17801][T1_DupTbLease][T1][YB42AC140005-000631D6BF8CB92A-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.358272] INFO [SQL.PC] inner_cache_evict (ob_ps_cache.cpp:749) [18779][T1_PsCacheEvict][T1][Y0-0000000000000000-0-0] [lt=26] ps cache evict(tenant_id=1, stmt_id_map_.size()=1, stmt_info_map_.size()=1, op.get_used_size()=1498, get_mem_high()=48318300, expired_stmt_ids.count()=0, closed_stmt_ids.count()=1) [2025-04-03 14:01:21.362114] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DF0-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.362193] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DF0-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1696, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.362887] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.362912] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=21] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28862 [2025-04-03 14:01:21.362951] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28862 [2025-04-03 14:01:21.362977] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.365889] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.365908] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=16] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28864 [2025-04-03 14:01:21.365948] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28864 [2025-04-03 14:01:21.365954] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.374642] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.379379] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660081379029]}) [2025-04-03 14:01:21.379644] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660081379412]}) [2025-04-03 14:01:21.379929] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660081379661]}) [2025-04-03 14:01:21.381343] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660081379944]}) [2025-04-03 14:01:21.381769] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.381791] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=17] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28866 [2025-04-03 14:01:21.381850] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28866 [2025-04-03 14:01:21.381859] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.382264] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B97-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.385694] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.387884] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B97-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.403056] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:21.404355] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B98-0-0] [lt=1] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.405329] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B98-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.406078] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:130) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.408372] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:184) [17829][T1_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=1] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:13286744064}, ls->get_ls_id()={id:1}) [2025-04-03 14:01:21.415740] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.415796] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=50][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.415850] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.415867] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=11][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.425650] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:21.428974] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DF6-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.429083] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DF6-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=863, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.449541] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DF7-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.466952] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4816381394, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:21.467901] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18641][T1004_TxLoopWor][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.472763] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DF9-0-0] [lt=0][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.472785] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DF9-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.474829] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=34][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.485780] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.486418] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=1] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:21.486624] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash time detail, (compute_wash_size_time=125, refresh_score_time=45, wash_time=6) [2025-04-03 14:01:21.488422] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B99-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.488557] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660081485436]}) [2025-04-03 14:01:21.488948] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=2314, clean_start_pos=1509936, clean_num=62914) [2025-04-03 14:01:21.489167] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B99-0-0] [lt=2] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.489503] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B99-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.489840] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B99-0-0] [lt=1] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.493270] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B99-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.493825] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4683907287, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:21.497026] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B99-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.504490] INFO [COMMON] replace_map (ob_kv_storecache.cpp:743) [17447][KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.505319] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B99-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1001, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.507216] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:325) [18288][T1003_Occam][T1003][YB42AC140005-000631D6BF9CDA6F-0-0] [lt=43] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.510063] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:140, src_addr:172.20.0.3:28876 [2025-04-03 14:01:21.515850] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.516634] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.516660] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.516674] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=10][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.517399] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.517413] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=9][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.518624] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17514][UserLocAsyncUp0][T1003][YB42AC140005-000631D6BD3F2B9B-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1003, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.522785] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.522926] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28880 [2025-04-03 14:01:21.522951] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17414][ServerGTimer][T1][YB42AC140005-000631D6BBFCD1EE-0-0] [lt=28][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.523556] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:679) [17414][ServerGTimer][T1][YB42AC140005-000631D6BBFCD1EE-0-0] [lt=0] ObKVCache try flush washable memblock details(ret=-4273, force_flush=false, tenant_id=1, cache_id=-1, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.523936] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17414][ServerGTimer][T1][YB42AC140005-000631D6BBFCD1EE-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.525689] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:21.529602] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DFE-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.529670] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3DFE-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2061, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.530596] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.530639] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=35] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28882 [2025-04-03 14:01:21.530739] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=38] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28882 [2025-04-03 14:01:21.548759] INFO [CLOG] run1 (ob_garbage_collector.cpp:1352) [17810][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.551912] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0] ====== tenant freeze timer task ====== [2025-04-03 14:01:21.552623] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=56][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.554313] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17810][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.554989] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1640) [17810][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=1] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-04-03 14:01:21.555159] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E00-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.556633] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.556671] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:552) [18521][T1004_Occam][T1004][Y0-0000000000000000-0-0] [lt=0] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=2048, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=49, Freeze TxData Trigger Memory(MB)=40, Total TxDataTable Hold Memory(MB)=55, Total TxDataTable Memory Limit(MB)=204) [2025-04-03 14:01:21.563187] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.563236] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28886 [2025-04-03 14:01:21.563322] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28886 [2025-04-03 14:01:21.563349] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=23] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.565561] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E01-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.565598] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E01-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=303, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.570770] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.572092] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=4816487125, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:21.572113] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:21.574916] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.575757] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.575805] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=40] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28888 [2025-04-03 14:01:21.575936] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=57] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28888 [2025-04-03 14:01:21.575964] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.576551] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E02-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.576598] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E02-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=871, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.578581] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:497) [17796][T1_TenantMetaMe][T1][Y0-0000000000000000-0-0] [lt=37] Recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3916EEE", sizeof(T):3952, used_obj_cnt:812, free_obj_hold_cnt:138, allocator used:3777200, allocator total:3916800}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:120, free_obj_hold_cnt:2, allocator used:8002102, allocator total:8002102}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=11918902, tablets_mem_limit=429496720, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):2368, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1920, used_obj_cnt:30, free_obj_hold_cnt:19, allocator used:97216, allocator total:130560}, pending_cnt=0, wait_gc_count=0, tablet count=932) [2025-04-03 14:01:21.582634] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:305) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=16] ====== traversal_flush timer task ====== [2025-04-03 14:01:21.582987] INFO [MDS] flush (mds_table_mgr.cpp:172) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=32] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x2b739dad0510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1}, ls_create_status:1, clog_checkpoint_scn:{val:1743616943687716218, v:0}, clog_base_lsn:{lsn:4227600384}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743618994719735223, v:0}, all_id_meta:{id_meta:[{limited_id:1743655491736477739, latest_log_ts:{val:1743655475118797847, v:0}}, {limited_id:22000001, latest_log_ts:{val:1743616321186417451, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:624, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:21.583118] INFO [MDS] flush (mds_table_mgr.cpp:195) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=87] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:200001}, rec_scn:{val:1743616894738777400, v:0}}, {tablet_id:{id:200002}, rec_scn:{val:1743616894738777401, v:0}}, {tablet_id:{id:200003}, rec_scn:{val:1743616894738777402, v:0}}, }, max_consequent_callbacked_scn={val:1743655277355487643, v:0}, *this={this:0x2b739db96510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1001}, ls_create_status:1, clog_checkpoint_scn:{val:1743616894738777400, v:0}, clog_base_lsn:{lsn:3489447936}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616894738777399, v:0}, all_id_meta:{id_meta:[{limited_id:1740557836670287627, latest_log_ts:{val:1740557820032434136, v:0}}, {limited_id:19000001, latest_log_ts:{val:1740413613052489362, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743590092673837460, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:637, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:21.583189] INFO [MDS] flush (mds_table_mgr.cpp:195) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=39] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1002}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:3, ordered_tablets:{{tablet_id:{id:200009}, rec_scn:{val:1743616823359480140, v:0}}, {tablet_id:{id:200006}, rec_scn:{val:1743616823359480141, v:0}}, {tablet_id:{id:200007}, rec_scn:{val:1743616823359480142, v:0}}, }, max_consequent_callbacked_scn={val:1743655265068231480, v:0}, *this={this:0x2b73ab388510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1002}, ls_create_status:1, clog_checkpoint_scn:{val:1743616823359480140, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616823359480139, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743590092742141696, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:1, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1002}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1002, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:21.583297] INFO [MDS] flush (mds_table_mgr.cpp:195) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=39] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1003}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:5, ordered_tablets:{{tablet_id:{id:1152921504606846977}, rec_scn:{val:1743616894738777400, v:0}}, {tablet_id:{id:200008}, rec_scn:{val:1743616894738777401, v:0}}, {tablet_id:{id:200005}, rec_scn:{val:1743616894738777402, v:0}}, {tablet_id:{id:1152921504606846978}, rec_scn:{val:1743616894840797864, v:0}}, {tablet_id:{id:200004}, rec_scn:{val:1743616894840797865, v:0}}, }, max_consequent_callbacked_scn={val:1743655479483236415, v:0}, *this={this:0x2b7375de2510, is_inited:true, freezing_scn:{val:0, v:0}, ls:{ls_meta:{tenant_id:1004, ls_id:{id:1003}, ls_create_status:1, clog_checkpoint_scn:{val:1743616894738777400, v:0}, clog_base_lsn:{lsn:0}, rebuild_seq:1, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1743659146207410023, v:0}, tablet_change_checkpoint_scn:{val:1743616894738777399, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1743614587358473541, v:0}, rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2, proposal_id:14, palf_env_:0x2b735c5c4030, is_in_stop_state_:false, is_inited_:true, id_:1003}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1003, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:5505024, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}}}) [2025-04-03 14:01:21.583348] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:336) [18621][T1004_Flush][T1004][Y0-0000000000000000-0-0] [lt=49] succeed to traversal_flush(ret=0, ls_cnt=4) [2025-04-03 14:01:21.584867] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=11] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.584901] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=27] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28890 [2025-04-03 14:01:21.584978] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28890 [2025-04-03 14:01:21.584989] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=8] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.585849] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=72][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.588093] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E03-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.588174] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E03-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1400, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.591854] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.591914] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660081591368]}) [2025-04-03 14:01:21.592332] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660081591938]}) [2025-04-03 14:01:21.592746] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660081592358]}) [2025-04-03 14:01:21.594650] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660081592768]}) [2025-04-03 14:01:21.595980] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=4684009721, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:21.596046] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=58] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:21.596539] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1004, gts_local_cache={srr:[mts=1743659610587347], gts:1743659610593488791, latest_srr:[mts=1743660081594962]}) [2025-04-03 14:01:21.599192] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17690][T1_Occam][T1][YB42AC140005-000631D6BCCCDA22-0-0] [lt=0][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.600057] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E04-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.601294] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17690][T1_Occam][T1][YB42AC140005-000631D6BCCCDA22-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.604593] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2B9C-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.615904] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=5][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.615959] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=48][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.615979] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=14][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.616028] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.621955] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E06-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.622065] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E06-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=858, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.625765] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:21.628546] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.628563] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28896 [2025-04-03 14:01:21.628627] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28896 [2025-04-03 14:01:21.628635] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=6] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.633357] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E07-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.633432] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E07-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1170, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.634514] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.634531] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=14] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28898 [2025-04-03 14:01:21.635481] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=911] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28898 [2025-04-03 14:01:21.635494] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=9] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.647681] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [17533][BlkMgr][T0][Y0-0000000000000000-0-0] [lt=2][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.647888] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E08-0-0] [lt=0] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.647970] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E08-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1676, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.651497] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:142) [17893][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-04-03 14:01:21.662250] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E09-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.662320] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E09-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=986, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.667373] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=4816582310, log_service_ts=1743655265084940, min_tx_service_ts=4611686018427387, timestamp={val:1743655265084940962, v:0}) [2025-04-03 14:01:21.667444] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=60] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655261104510952, v:0}", last_lsn={lsn:13483016964}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:21.673656] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E0A-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.673738] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E0A-0-0] [lt=0] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=1239, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.674989] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18305][T1003_LogLoop][T1003][Y0-0000000000000000-0-0] [lt=105][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.675922] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=4] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.675983] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=37] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28900 [2025-04-03 14:01:21.676073] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28900 [2025-04-03 14:01:21.676103] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=10] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.682284] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=7] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.682312] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=24] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.684400] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=0] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:21.685923] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [17701][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.689388] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1154) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=true, sys_total_wash_size=4817199104, global_cache_size=106119168, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[500, 508, 509, 1, 1001, 1002, 1003, 1004]) [2025-04-03 14:01:21.689649] INFO [COMMON] wash (ob_kvcache_store.cpp:341) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash time detail, (compute_wash_size_time=160, refresh_score_time=55, wash_time=8) [2025-04-03 14:01:21.692944] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:636) [17446][KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.693968] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:184) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33] get wrs ts(ls_id={id:1}, delta=4684108387, log_service_ts=1743655397585394, min_tx_service_ts=4611686018427387, timestamp={val:1743655397585394700, v:0}) [2025-04-03 14:01:21.694012] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:214) [18030][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1743655392997335048, v:0}", last_lsn={lsn:28444645143}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-04-03 14:01:21.698068] WDIAG [LIB] ob_malloc (ob_malloc.h:39) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4013] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.698118] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=1743659480945777], gts:1743659480947168402, latest_srr:[mts=1743660081697070]}) [2025-04-03 14:01:21.698131] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:171) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", gts_tenant_info={v:1}) [2025-04-03 14:01:21.698270] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:196) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E0C-0-0] [lt=1] update sys_tenant's ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-04-03 14:01:21.698353] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [17484][LSSysTblUp0][T0][YB42AC140005-000631D6CABF3E0C-0-0] [lt=1] update ls replica(ret=-4013, ret="OB_ALLOCATE_MEMORY_FAILED", time_used=2693, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"172.20.0.5:2882", sql_port:2881, role:2, member_list:[{server:"172.20.0.3:2882", timestamp:1}, {server:"172.20.0.4:2882", timestamp:1}, {server:"172.20.0.5:2882", timestamp:1743648235302292}], replica_type:0, proposal_id:159, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:3, zone:"zone3", paxos_replica_number:3, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-04-03 14:01:21.698400] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72f767e270 fd=93 [2025-04-03 14:01:21.698416] INFO ussl_on_accept (ussl_listenfd.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=12] [ussl] accept new connection, fd:93, src_addr:172.20.0.3:28906 [2025-04-03 14:01:21.698454] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=5] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28906 [2025-04-03 14:01:21.698475] INFO dispatch_accept_fd_to_certain_group (group.c:682) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=19] PNIO dispatch fd to certain group, fd:93, gid:0x100000000 [2025-04-03 14:01:21.699194] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=1743659480925871], gts:1743659480926054981, latest_srr:[mts=1743660081698147]}) [2025-04-03 14:01:21.699694] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=1743659480926089], gts:1743659480926263727, latest_srr:[mts=1743660081699221]}) [2025-04-03 14:01:21.701383] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:580) [17556][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1003, gts_local_cache={srr:[mts=1743659955660404], gts:1743659955695473921, latest_srr:[mts=1743660081699711]}) [2025-04-03 14:01:21.706738] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18046][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.711133] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18111][T1002_LogUpdate][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.716088] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=12][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.716152] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=57][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1002, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.716176] INFO [PALF] check_parent_health (log_config_mgr.cpp:2824) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=16] re_register_parent reason(palf_id=1001, self="172.20.0.5:2882", is_registering_timeout=true, first_registration=false, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=1743660077710431, last_first_register_time_us=1743648246815801, register_time_us=1743660077710431, curr_time_us=1743660081716174) [2025-04-03 14:01:21.717751] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1001, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.717799] WDIAG [PALF] get_election_leader_without_lock_ (palf_handle_impl.cpp:4609) [18499][T1004_LogLoop][T1004][Y0-0000000000000000-0-0] [lt=39][errcode=-4209] election has no leader(ret=-4209, this={palf_id:1003, self:"172.20.0.5:2882", has_set_deleted:false}) [2025-04-03 14:01:21.719101] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18710][PxTargetMgr0][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.720743] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:346) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=1] ====== check clog disk timer task ====== [2025-04-03 14:01:21.720833] INFO [PALF] get_disk_usage (palf_env_impl.cpp:890) [18238][T1002_CKClogDis][T1002][Y0-0000000000000000-0-0] [lt=46] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.725832] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:147) [18617][T1004_STSWorker][T1004][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-04-03 14:01:21.731162] INFO ussl_eloop_regist (ussl_eloop.c:39) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] sock regist: 0x2b72e6174920 fd=93 [2025-04-03 14:01:21.750280] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [18638][T1004_HAService][T1004][Y0-0000000000000000-0-0] [lt=32] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.750712] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [17514][UserLocAsyncUp0][T1001][YB42AC140005-000631D6BD3F2BA5-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456) [2025-04-03 14:01:21.750865] INFO [CLOG] run1 (ob_garbage_collector.cpp:1352) [18596][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=0] Garbage Collector is running(seq_=1179, gc_interval=10000000) [2025-04-03 14:01:21.751022] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [18596][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.751715] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18596][T1004_GCCollect][T1004][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.756391] INFO acceptfd_handle_first_readable_event (handle-event.c:440) [17462][ussl_loop][T0][Y0-0000000000000000-0-0] [lt=1] [ussl] auth mothod is NONE, the fd will be dispatched, fd:93, src_addr:172.20.0.3:28914 [2025-04-03 14:01:21.761692] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:131) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.762853] WARN common_alloc (ob_tenant_ctx_allocator.cpp:464) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4013] No memory or reach tenant memory limit([OOPS]="alloc failed reason", msg=server memory has reached the upper limit(server_hold: 10735362048, server_limit: 10737418240, alloc_size: 2097152)) [2025-04-03 14:01:21.762888] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:552) [18133][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0] [TenantFreezer] Trigger Tx Data Table Self Freeze(Tenant Total Memory(MB)=512, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=33, Freeze TxData Trigger Memory(MB)=10, Total TxDataTable Hold Memory(MB)=36, Total TxDataTable Memory Limit(MB)=51) [2025-04-03 14:01:21.767744] WDIAG print_tenant_memblock_info (ob_kvcache_store.cpp:790) [17828][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-04-03 14:01:21.768286] INFO [COMMON] try_flush_washable_mb (ob_kvcache_store.cpp:660) [18596][T1004_GCCollect][T1003][YB42AC140005-000631D6C42CCD3B-0-0] [lt=0] can not find enough memory block to wash(ret=-4273, size_washed=0, size_need_washed=6291456)